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Chapter  1 

Introduction 


1.1  Telerobotic  Surgery 

With  the  development  of  modem  technologies  (including  remote  sensing,  robotics,  medical 
informatics,  telepresence,  supercomputing,  digital  image  processing  and  tele¬ 
communications),  telemedicine  has  become  a  very  active  research  area  in  the  1990s.  The 
original  idea  of  telemedicine  was  to  provide  medical  care  over  long  distance  through  the  use 
of  telecommunications.  In  the  early  days  of  the  1960s,  the  primary  target  was  to  implement 
telediagnoses  [Sheridan,  1992].  Now  efforts  are  underway  in  the  world  to  develop 
telesurgery,  where  a  surgeon  operates,  by  means  of  a  telerobotic  device  and  a 
communications  link,  on  a  patient  who  is  an  arbitrary  distance  away.  The  current  goal  in 
the  U.S.A.  is  the  development  of  a  telerobotic  surgical  system  that  will  assist  a  medical  care 
provider  (who  might  have  limited  medical  care  experience)  in  the  management  of  a  medical 
emergency  (including  surgical  intervention)  at  a  location  that  is  remote  from  the  hospital 
and  the  experts  available  there.  Such  an  event  might  occur  in  both  civilian  and  non-civilian 
situations  in  locations  such  as  remote  rural  areas,  ships  at  sea,  battle  fields,  natural 
disasters,  research  stations  in  the  Antarctic,  and  NASA  space  missions,  among  others. 

Telesurgery  is  a  special  application  of  teleoperation  that  requires  good  telerobotic  devices 
to  perform  the  remote  operations  and  a  good  quality  of  telepresence  (visual,  audio,  and 
haptic  feedback)  to  enhance  the  surgeon’s  medical  performance.  So  far  the  research 
projects  on  telesurgery  have  had  different  focuses,  such  as  surgical  telerobot  design, 
surgical  tool  settings,  telepresence  surgery,  etc.  Experiments  on  surgical  telerobotics  from 
1992  until  today  have  evolved  considerably  [Kanade  et  al  1994].  Green[1992,1995],  at 
SRI  International  in  California,  has  developed  an  advanced  telesurgery  system,  which  is  a 
two  armed,  five  degree  of  freedom,  bilateral  force  reflecting  telerobotic  system.  Through 
the  demonstration  (called  as  Telepresence  Laparoscopic  Surgery)  in  Washington,  D.C.  at 
the  annual  convention  of  the  Association  of  the  U.S.  Army,  in  1994,  SRI  established  that 
precise  surgical  procedures  can  be  carried  out  with  telepresence.  In  1994  another  group  at 
Johns  Hopkins  Bay  view  Medical  Center  conducted  an  experiment  on  telerobotic 
laparoscopic  surgery  for  the  purpose  of  assessing  the  feasibility  of  telerobotic  assisted 
surgery  [Kavoussi,  1994].  These  experiments  did  not  involve  significant  time  delay. 
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The  first  experiment  of  telerobotic  surgery  in  the  world  was  carried  out  between  the 
NASA  Jet  Propulsion  Laboratory  in  Pasadena,  California,  and  the  Telerobotic  Laboratory 
of  the  Politenico  di  Milano,  Italy  on  July  7,1993  [Rovetta,1996].  During  that  experiment, 
an  Italian  robot  in  Milan  was  remotely  controlled  by  an  Italian  surgeon  in  the  United  States. 
The  surgical  operation  involved  the  execution  of  a  biopsy,  aspiration  of  organic  material, 
and  two  incisions  in  preparation  of  laparoscopy.  Transmission  was  effected  by  means  of  a 
double  satellite  link,  with  three  transceiver  stations:  one  in  Italy,  one  close  to  New  York, 
and  one  in  Pasadena,  and  two  geostationary  satellites,  the  first  over  the  Atlantic  and  the 
second  over  the  United  States.  The  route  length  of  the  signals  was  150,000  km  in  each 
direction  and  the  transmission  delay  time  was  significant  (1.2  seconds)  on  the  satellite 
network.  To  increase  safety  and  system  reliability,  an  optical  fiber  network  with  ISDN 
located  on  the  ground  and  on  the  sea  bed  was  used  to  obtain  a  parallel  signal  transmission 
with  fast  transmission  rate.  This  experiment  showed  that  the  time  delay  in  telerobotics 
should  always  be  taken  into  consideration  so  that  surgical  telerobotics  can  be  safely  applied 
in  those  operations  where  time  interaction  is  kept  under  control  (e.g.  in  biopsies  and 
incisions  under  anesthesia).  While  these  experiments  involved  time  delay,  they  did  not 
involve  force  feedback  in  the  telerobotic  systems. 

Sheridan  [1994]  suggested  to  perform  telesurgery  by  using  the  available  intelligence  and 
assistive  dexterity  in  the  form  of  a  paramedic  (or  in  some  cases  an  untrained  assistant) 
available  locally  to  the  patient,  which  suggests  a  more  acceptable  application  of  telesurgery 
in  the  future.  Hence  cooperative  manipulation  (supervisory  control)  between  a  paramedic 
local  to  the  patient  and  a  surgeon  through  a  telerobot  becomes  an  important  issue  in 
telesurgery  in  the  scientific  sense.  Besides,  time  delay  effects  in  telesurgery,  which  degrade 
the  quality  of  telepresence  (visual,  audio,  and  force  feedback)  are  indeed  a  very  important 
issue. 

Previous  experiments  on  telesurgeiy  have  not  paid  much  attention  to  the  combination  of 
time  delay  and  telepresence  (especially  force  feedback).  In  this  thesis,  teleoperation 
including  force  feedback  with  time  delay  (which  is  called  stable  force  reflecting  control  with 
time  delay)  will  be  addressed. 

1.2  Telepresence  in  Telesurgery 

Telesurgery  is  a  natural  extension  of  developments  in  teleoperation,  where  the  latter  is 
defined  as  “the  extension  of  human  inspection  and  manipulation  capability  to  remote  or 
otherwise  inaccessible  locations”[Sheridan,  1996].  A  telerobot  is  a  device  that  has  sensors 
(such  as  video  camera)  and  actuators  (hands  for  grasping  and  arms  for  positioning  and 


conveying  forces  to  the  hands,  plus  some  means  of  mobility),  which  are  combined  with  a 
means  of  communicating  information  to  and  from  the  human  operator.  Usually,  a  complete 
telerobotic  system  consists  of  master-slave  robotic  manipulators,  communication  link, 
remote  sensing  devices  (including  video,  audio,  and  force  feedback  etc.)  (Figure  1-1). 
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Figure  1-1:  Block  diagram  of  a  telerobotic  system 

In  most  telerobotic  systems,  human  operators  control  the  endpoint  of  the  master 
manipulator  (robot),  which,  in  turn,  determines  the  joint  angles  of  the  master  and  the  slave 
manipulators.  In  a  force-reflecting  teleoperator  system,  the  reaction  force  sustained  by  the 
slave  is  fed  back  and  reconstructed  by  the  master  manipulator  to  provide  the  operator  a 
sense  of  what  he  would  feel  if  he  were  handling  the  task  directly.  The  kinesthetic  coupling 
between  the  human  operator  and  the  task  is  part  of  the  currently  accepted  idea  of 
“telepresence”  which  implies  feeding  back  all  sensory  information  and  energy,  including 
vision,  sound,  smell,  force,  tactility  etc.,  such  that  the  human  operator  feels  physically 
present  at  the  remote  site. 

1.2.1  Telepresence  for  better  operation 

Telepresence  is  commonly  claimed  to  be  important  for  direct  manual  telemanipulation.  It 
has  yet  to  be  shown  how  important  is  the  sense  of  “feeling  present”  per  se  is  to 
performance  as  compared  to  simply  having  high  resolution,  a  wide  field  of  view  and  other 
attributes  of  good  sensory  feedback  [Sheridan,  1992].  Generally  speaking,  telepresence 
can  be  classified  as  visual  telepresence,  auditory  telepresence  (binaural  localization  and 
spectral  correspondence  to  the  real  world),  force  (muscle  force)  telepresence,  and  tactile 
(skin  sense)  telepresence. 

The  quality  of  telepresence  has  direct  effects  on  the  performance  of  teleoperation.  From 
the  experience  that  a  force-reflecting  teleoperator  system  performs  better  than  a  system 
without  force  feedback  one  can  understand  the  concept  of  telepresence  partly.  It  is 
intuitively  assumed  that  the  performance  of  a  remote  manipulator  will  rival  direct  manual 
operation  if  the  operator  is  presented  with  all  sensory  information  of  the  remote  site  and  if 
the  remote  manipulator  has  the  same  dexterity  as  a  human  arm  (or  hand).  Presumably  under 


such  circumstances,  the  human  operator  can  feel  as  if  he  is  at  the  remote  site  and  is 
performing  the  operation  directly. 

However,  the  concept  of  “to  feel  as  if  at  the  remote  site”  is  far  beyond  the  necessary 
requirement  of  the  original  purpose  of  using  teleoperation  systems  to  perform  tasks  in  an 
inaccessible  location.  For  example,  in  a  combat  environment  the  operator  might  be  subject 
to  a  high  level  of  stress  and  anxiety  if  he  were  presented  with  the  realistic  reconstruction  of 
the  force,  sound,  smell  and  sights  of  the  battle  field.  In  such  cases,  “to  feel  as  if  at  the 
remote  site”  is  not  appropriate  since  the  operator  will  suffer  both  physically  and  mentally 
from  the  hazardous  environment  [Blais  &  Lyons,1988].  On  the  other  hand,  selected  and 
modified  feedback  may  reduce  the  stress  of  fatigue  associated  with  a  task  and  increase  the 
performance  level.  Since  “to  feel  as  if  at  the  remote  site”  is  not  always  helpful  in 
teleoperation,  the  ideal  telepresence  should  not  be  to  feed  back  all  sensory  information  and 
energy  of  the  remote  site  the  operator,  but  should  be  to  feed  back  such  information  and 
energy  as  will  increase  the  performance  level,  but  not  such  information  and  energy  that  will 
jeopardize  the  operation.  In  conclusion,  teleoperation  should  be  a  means  to  achieve  higher 
performance  with  necessary  remote  information,  but  not  an  end  in  itself. 

Force  reflection,  considered  as  a  subset  of  telepresence,  has  been  shown  to  be  beneficial 
in  remote  constrained  motion  tasks  [Sheridan,  1992].  Task  completion  time  is  often 
reduced  by  as  much  as  40  percent  when  the  operator  is  given  force  feedback  information. 
Even  with  visual  feedback,  human  performance  in  terms  of  task  completion  time  is  still 
severely  limited  unless  the  operator  is  supplied  with  force  feedback.  Therefore,  force 
reflection  capability  has  been  considered  necessary  for  a  high  performance  teleoperator 
system.  However,  exact  force  reflection,  just  like  other  items  that  constitute  telepresence  is 
not  always  necessary  or  most  helpful.  The  following  section  discusses  the  problems  of 
force  feedback. 

1.2.2  The  limits  of  force  reflecting  teleoperation 

Since  the  human  muscle-skeletal  system  is  directly  coupled  to  the  teleoperator  and  task 
dynamics  in  force  reflection  teleoperation,  the  overall  system  dynamics  becomes  very 
complex  and  the  performance  of  the  overall  system  is  very  much  dependent  upon  the 
human  capability.  Hence,  the  performance  of  the  remote  manipulation  can  be  affected  in 
several  ways: 

(1)  A  force  reflection  teleoperator  system  can  be  much  more  fatiguing  than  a  positioning 
device  such  as  a  mouse  or  a  teleoperator  without  force  feedback.  Since  the  operator  is 
coupled  with  the  task  in  a  force  reflection  system,  whatever  force  is  exerted  by  the  slave 


manipulator  will  be  reflected  to  the  operator  and  sustained  by  the  operator’s  muscles. 
Therefore,  it  is  often  fatiguing  to  use  a  force  reflective  teleoperator  system. 

(2)  If  the  teleoperator  exactly  reflects  the  environment  to  the  human  operator  (complete 
transparency  of  the  interface),  then  the  capability  of  the  system  will  be  exactly  limited  by 
the  human  operator  as  well.  For  example,  the  remote  manipulator  will  not  be  able  to  lift  a 
weight  heavier  than  what  the  operator  can  lift  directly,  or  execute  a  fine  motion  more 
precisely  than  what  he  can  execute  directly.  Moreover,  if  a  task  requires  an  input  bandwidth 
higher  than  what  the  human  operator  can  exert,  the  operator  will  not  be  able  to  do  it. 

(3)  The  dynamics  and  control  of  a  force-reflection  teleoperator  is  very  complex.  The 
force  reflection  will  develop  a  closed-loop  between  the  human-master  subsystem  and  the 
slave-task  subsystem.  The  dynamics  of  a  force  reflection  teleoperator  system  not  only 
depends  on  the  above  two  parts,  but  also  depends  on  the  transmission  between  the  two 
parts.  For  example,  the  stability  problems  due  to  noise  and  signal  transmission  time  delay 
are  well  acknowledged  and  have  severely  limited  the  remoteness  of  the  operation  [Sheridan 
1992].  Because  of  the  reflection  nature  of  force  feedback,  delayed  force  feedback,  unlike 
delayed  visual  feedback  and  audio  feedback,  could  interfere  with  the  operator’s  command 
input.  Therefore,  Ferrell  [1966]  stated  that  force  feedback  is  “not  only  a  source  of 
information,  but  may  act  as  a  disturbance  input  as  well”. 

To  summarize,  although  the  performance  level  measured  by  task  completion  time  can  be 
remarkable  in  certain  types  of  tasks,  a  force  reflecting  teleoperator  system  is  fatiguing,  is 
limited  by  the  dexterity  and  capacity  of  the  human  operator,  and  can  be  unstable  and  hard  to 
control  especially  in  the  presence  of  time  delay.  Even  with  an  ideal  teleoperator  system 
which  provides  a  complete  transparent  interface  between  the  operator  and  the  task,  the 
performance  level  is  still  limited  by  the  performance  level  of  the  human  himself. 


1.3  Stable  force  reflection  with  time  delay 

1.3.1  Time  delay  effect  on  force  reflection  of  telerobotic  surgery 

Several  successful  demonstrations  of  remotely  controlled  endoscopic  surgery  and 
laparoscopic  surgery  have  taken  place  recently  [Green, 1995;  Kavoussi,1994;  Partin,1995]. 
However,  in  these  demonstrations  there  was  no  appreciable  time  delay  between  the 
surgeon’s  actions  and  the  resulting  laparoscopic  image  or  between  the  surgeon’s 
actions  and  the  force  feedback.  Rovetta’s  demonstration  of  telesurgery  between  Milan,  Italy 
and  Pasadena,  CA,  U.S.A.  used  satellite  communication  networks  which  took  1.2  seconds 
time  delay  for  one-way  information  transmission  (including  video,  audio,  and  sensing 


signals )  but  no  force  feedback.  In  1993,  Yamashita  demonstrated  his  teleoperation  system 
between  Tokyo,  Japan  and  Encina,  CA,U.S.A.  via  ISDN  communication  lines  across  the 
Pacific  Ocean  [Yamashita,  1993].The  data  transmission  time  was  approximately  0.70 
seconds  which  doesn’t  include  the  necessary  time  delay  for  image  compression  and 
decompression  (minimum  0.60  seconds).  The  latest  available  technology  to  do  video 
conference  typically  use  several  ISDN  channels  or  T1  link  that  result  in  delays  in  the  video 
signals  of  at  least  a  half  second,  round  trip  (e.g.  0.60  seconds  for  the  Zydacron  system 
[Zydacron,96]).  Therefore,  time  delays  always  exist  in  the  teleoperation  systems  because 
of  the  constraint  of  the  available  technology. 

In  the  case  of  telesurgeiy,  (particularly  where  satellites  and  long  distance  ISDN  lines  are 
used)  the  time  delays  appear  both  in  the  transmission  of  commands  to  the  remote  surgical 
system  and  in  the  feedback  of  information  (e.g.  video,  audio )  to  the  surgeon.  Among  all 
the  effects  of  time  delays,  such  as  image  transmission,  control  signal  transmission,  sensing 
information  feedback,  and  so  on,  force  reflection  is  dominantly  important  to  perform 
practical  telesurgery .  The  latter  is  true  because  a  human  can  ignore  visual  time  delays  of  a 
small  fraction  of  a  second,  but  humans  in  a  force  loop  inherently  close  the  loop  through  the 
control  handle  and  force  feedback  generating  spurious  movements  that  cannot  be 
suppressed  by  the  human  operator  without  holding  the  control  handle  rigid. 

The  force  reflection  control  of  a  teleoperation  system  with  large  time  delay  ( more  than 
0.2  seconds )  becomes  unstable  [Bejczy,1994].  Usually  the  communication  time  delay  is  in 
order  of  1  second  if  satellite  communication  is  used  [Sheridan,  1989].  When  the 
communication  time  delay  become  significantly  large  comparing  with  the  dynamic  time 
constants  in  the  robotic  system  (e.g.  time  delay  >0.2  seconds),  the  delayed  forces  imposed 
on  the  operator’s  (surgeon’s)  hand  will  be  significantly  out-of-phase  with  his  or  her 
intended  motions,  and  will  therefore  excite  instability  in  the  teleoperation  system  and  make 
teleoperation  or  telesurgery  very  difficult  to  implement.  On  the  other  hand,  from  a  control 
system  point  of  view,  large  time  delay  generates  a  large  phase  lag  in  the  system  loop, 
decreasing  the  phase  margin  of  the  system  significantly  and  making  the  system  unstable.  It 
is  very  difficult  to  design  a  compensator  for  such  a  delay  [Niemeyer  and  Slotine,  1991]. 
Therefore,  the  classic  linear  controller,  such  as  proportional-plus-derivative  (PD)  control 
can  not  make  the  teleoperation  system  with  large  time  delay  stable  unless  some  way  can  be 
found  to  change  the  system’s  physical  parameters  [Anderson,  1989].  A  better  way  to 
explain  the  instability  of  force  reflection  of  teleoperation  with  time  delay  is  to  apply  passive 
theory  or  scattering  theory,  which  will  be  described  in  the  thesis. 


1.3.2  Previous  works  on  stable  force  reflecting  control 

Several  approaches  to  coping  with  time  delay  in  telerobotics  have  been  proposed  since  the 
1960s.  The  previous  approaches  can  be  classified  into  the  following  different  types:  (1) 
visual  aiding  approaches  (including  “move  and  wait”  strategy,  predictor  display,  force 
signal  display),  (2)  passive  control  and  (3)  supervisory  control.  Ferrell  [1966],  Buzan 
[1989]  and  Buzan  and  Sheridan  [1989]  investigated  the  special  case  of  delayed  force 
feedback,  and  developed  specialized  predictors  to  cope  with  delayed  force.  Time  delay  in  a 
Teleoperation  loop  can  also  be  circumvented  altogether  by  using  supervisory  control,  in 
which  the  telerobot  is  programmed  to  perform  operations  in  short  closed  loop  segments 
and  control  is  managed  locally  through  artificial  sensors  and  a  computer  within  the  task 
segments  [Ferrell  and  Sheridan,  1967;  Brooks,  1979;  Conway  et  al.,  1990;  Funda  et  al. 
1992;  Sheridan,  1992].  Hirzinger  et  al.  [1993]  demonstrated  telerobotic  control  from  the 
Earth  to  the  NASA  Apace  Shuttle  for  simple  manipulation  tasks,  making  use  of  both 
predictor  display  and  supervisory  control  techniques.  Other  investigators  have  suggested 
means  of  artificially  damping  or  smoothing  control  signals  to  prevent  instability  which  can 
be  called  a  class  of  passive  control  [Raju,1986;  Anderson,  1988]. 

Raju  [1986,1988]  was  the  first  to  apply  two-port  network  models  to  the  stability 
analysis  of  teleoperators.  He  suggested  that  if  the  teleoperator  system  is  designed  such  that 
its  impedance  characteristics  can  be  adjusted  by  the  human  operator  for  different  tasks,  it 
may  be  possible  to  improve  the  performance  level  of  the  operator  using  both  system 
characteristics  equations  and  positive  real  matrices.  He  analyzed  and  solved  the  ‘bound’  of 
parameters  for  stabilizing  PD-controllers  against  all  passive  environments.  The  stable  port 
behavior  can  thus  be  specified  using  gains  within  the  parameter  bounds.  Hannaford  & 
Spong  [1988]  introduced  a  similar  analysis  using  hybrid  two-port  networks. 

Anderson  and  Spong  [1988]  extended  the  methodology  of  teleoperator  network  design 
with  the  passivity  theory  and  the  scattering  matrix.  Based  on  the  passivity  theorem,  overall 
system  stability  is  maintained  by  securing  the  passivity  of  each  subsystem.  In  this  way,  the 
design  process  is  much  simplified.  However,  the  trade-off  is  that  the  overall  system 
becomes  too  conservative  (more  stable  than  needed).  By  designing  the  communication 
subsystem  with  lossless  scattering  matrices,  Anderson  and  Spong  also  solved  the  stability 
aspect  of  the  time  delay  problem.  Niemeyer  and  Slotine  [1991]  presented  a  similar  result  in 
dealing  with  time-delay  problems.  Based  on  Raju’s  work.  Chin  analytically  derived  the 
design  guideline  for  stable  passive  controllers  [Chin,  1991]. 

With  the  inspiration  of  the  “move  and  wait”  strategy,  and  the  function  of  the  series 
elastic  actuator  [Pratt,  1995],  the  author  proposed  a  novel  control  approach— Fuzzy  Sliding 
Control.  In  this  thesis,  the  new  control  approach  will  be  described  in  detail. 
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1.4  Outline  of  Thesis  Contents 


This  thesis  introduces  my  contributions  to  the  subject,  stable  force  reflecting  control  of 
telerobotic  surgery  system  with  time  delay.  The  main  part  of  the  thesis  focuses  on 
development  of  the  new  approach  of  Fuzzy  Sliding  Control  and  the  derivation  of  the 
modified  passive  control  approach.  Both  approaches  have  been  applied  in  a  telerobotic 
surgery  system.  This  thesis  is  organized  as  follows: 


Chapter  1 
Chapter  2 
Chapter  3 

Chapter  4 
Chapter  5 

Chapter  6 
Chapter  7 

Chapter  8 


introduces  the  technologies  in  telerobotic  surgery,  time  delay  effects  on 
force  reflection  of  teleoperation  and  the  previous  work  on  the  subject, 
explains  why  Fuzzy  Sliding  Control  (FSC)  is  useful  in  teleoperation,  what 
is  FSC  and  how  to  design  FSC. 

describes  the  system  structure  of  FSC  in  telerobotic  systems  and  the 
modeling  of  a  teleoperation  system,  and  it  elaborates  how  FSC  stabilizes 
the  teleoperation  system. 

presents  the  methodologies  of  passive  control  and  supervisory  control.  A 
modified  approach  to  passive  control  is  proposed. 

describes  the  experimental  studies  on  FSC  by  means  of  a  virtual 
environment.  Subjective  experiments  for  different  controllers  are  conducted 
with  respect  to  different  time  delays  (0-1  sec.).  Both  subjective  and 
objective  evaluation  are  described. 

introduces  system  design  of  a  telerobotic  system  including  hardware  and 
software  design  for  experimental  study  and  surgery  demonstration, 
presents  the  experimental  data  for  different  control  approaches,  e.g.  passive 
control  and  FSC.  The  resulting  analyses  are  based  on  the  dynamics 
responses. 

summarizes  this  thesis  and  discusses  the  direction  of  future  research. 


Chapter  2 

Theory  and  Design  of  Fuzzy  Sliding  Control 

2.1  Intuition  of  force  reflecting  control  with  time  delay 

From  our  simulations  and  experiments,  we  observed  that  when  the  remote  robot  (slave) 
makes  contact  with  the  task  environment,  due  to  the  delayed  information  feedback,  the 
master  robot  excites  a  system  oscillation  (instability).  This  instability  shows  less  of  a 
repetitive  pattern,  but  finds  a  self-sustaining  oscillation  at  a  frequency  inversely 
proportional  to  the  delay.  In  both  cases  the  instability  was  provoked  by  an  impulse 
disturbance  which  is  associated  with  the  slave  contact  with  the  environment,  and  the  action 
of  the  human  arm  (also  human  arm  locomotion  effects)  to  a  step  function  force  signal. 

Time  delay  has  the  following  effects  on  the  force  feedback  (or  force  telepresence ). 

•  The  human  operator  cannot  predict  exactly  the  instant  when  the  slave  robot  contacts 
with  the  task  environment; 

•  Once  the  slave  contacts  the  task  environment,  the  force  signal  in  the  slave  during  the 
transition  period  (from  free  motion  to  full  contact)  is  roughly  a  step  function; 

'  •  When  the  slave  force  signal  (step  function),  which  is  not  a  smooth  function  is 
reconstructed  on  the  master  robot,  the  human  operator  will  generate  an  impulse  (force 
&  position)  disturbance. 

Intuitively  there  are  several  ways  to  avoid  instability,  which  are: 

•  Visual  predictor  display.  Some  people  have  already  investigated  this  approach,  which 
indeed  improved  the  operation  performance.  However,  in  complex  task  cases,  it  is  very 
difficult  to  achieve  perfect  predictor  display  (predicted  graphics  overlaid  on  the  video 
image). 

•  Add  artificial  damping  to  the  robot.  This  gives  us  an  approach  called  passive  control. 
Using  the  artificial  damping  factors,  the  energy  injected  by  the  human  arm  in  the  form 
of  impulse  force  when  contact  takes  place  at  the  remote  site  can  be  dissipated  in  the 
master-slave  global  system. 

•  Add  a  “smart”  filter”  to  smooth  the  contact  force  signal  of  the  slave  robot  and  feed 
back  the  steady  force  signals  to  the  master  robot.  In  the  mean  time,  add  robustness  to 
the  controllers  of  both  slave  and  master  robot. 

For  the  third  approach,  there  are  some  difficulties  to  design  such  a  smart  filter  and  robust 
controllers,  because  there  are  so  many  uncertainties  in  the  telerobotic  system.  For  example, 
the  task  environment  may  vary  during  the  operations,  and  the  operator’s  arm  model  is  not 


known  to  us  even  today.  Without  precise  mathematic  models,  how  can  we  design 
appropriate  controllers  for  the  robots  ?  The  answer  is  that  we  can  build  the  model  from  our 
experience  and  control  engineer’s  expertise.  We  have  already  known  that  a  series-elastic 
actuator  can  have  very  good  stability  in  dealing  with  surface  contact  [Pratt,  1995].  If  we 
can  build  a  smart  control  that  has  adjustable  stiffness  and  robustness,  the  system 
performance  will  be  improved  significantly.  A  new  control  —Fuzzy  Sliding  Control  which 
has  the  two  features  (smart  filter,  adjustable  stiffness  and  robustness)  will  be  introduced  in 
the  following  sections. 


2.2  Fuzzy  Logic  and  Fuzzy  Control 

Fuzzy  logic  is  much  closer  in  spirit  to  human  thinking  and  natural  language  than  traditional 
logic.  Essentially  it  provides  an  effective  means  of  capturing  the  approximate,  inexact 
nature  of  the  real  world.  A  fuzzy  logic  controller  (FLC)  is  a  set  of  linguistic  control  rules 
which,  in  combination,  result  in  an  algorithm  to  map  quantitative  inputs  to  quantitative 
outputs.  The  methodology  of  the  FLC  is  very  useful  when  the  processes  are  too  complex 
for  analysis  by  conventional  quantitative  techniques  or  when  the  available  source  of 
information  can  only  be  interpreted  qualitatively,  inexactly,  or  uncertainly.  Thus  fuzzy  logic 
control  may  be  viewed  as  a  step  toward  a  rapprochement  between  conventional  precise 
mathematical  control  and  human-like  decision  making. 

Here  is  an  example  of  FLC  with  approximate  inference  (Mamdani’s  Approach) 
[Mamdani,  1976].  If  there  exist  N  rules  in  the  fuzzy  control  system,  the  fth  rule  is  in  the 
form  of 

IF  inputs  Xj  =  Ai ,  x2  =  Bj ,  THEN  u  =  Q  . 

Let  Xio,  x2o  be  the  inputs  of  xj,  x2  correspondingly,  so  the  truth  value  of  the  premise  (or 
membership)  can  be  expressed  as 
Mi  =  !Ui(xio )  a  MBi(x2o) 

where  MAi(*io  )  is  the  membership  of  Xi0  pertaining  to  fuzzy  set  A;  and  p.Bi(x2o), 
membership  of  x20  pertaining  to  Bi;  and  “a”  is  an  operator,  which  can  be  defined  as 
“minimum”  or  other  fuzzy  operator  [Pedrycz,  1993].  Then,  the  membership  of  the  control 
output  is 


where  “v”  is  an  operator,  which  can  be  defined  as  “maximum”  or  other  operators  [Pedrycz, 
1993].  Therefore  the  output  (fuzzyfication,  or  fuzzy  decision  making)  can  be  obtained 
from  the  centroid  formula. 
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The  inference  process  can  be  described  in  the  following  figure. 


Fuzzy  Decision:  uo 


Figure  2-1:  Inference  process  of  fuzzy  control 

Fuzzy  logic  control  has  been  applied  to  many  industrial  systems  successfully  since 
Mamdani’s  pioneering  work  in  1976  [Mamdani,  1976].  As  a  heuristic  approach,  fuzzy  logic 
control  is  usually  based  on  expert  knowledge.  For  an  ill-defined  and  complex  system,  fuzzy 
control  has  advantages  over  the  traditional  approach  because  it  is  not  necessary  to  do  the 
mathematic  modelling.  On  the  other  hand,  it  is  suitable  for  human  knowledge  interfacing 
because  of  linguistic  flexibility.  Although  fuzzy  control  is  very  successful,  especially  for  the 


control  of  nonlinear  systems,  there  is  a  lack  of  appropriate  design  technique  with  regard  to 
the  control  performance  and  stability  of  systems.  For  example,  fuzzy  control  is  rule  based, 
and  the  rules  chosen  by  the  human  experts  may  vary  for  a  given  system  plant.  The  tuning  of 
the  rule  base  is  done  in  a  trial  and  error  manner.  There  is  no  consistent  methodology  to 
guide  fuzzy  partition,  rule-tuning,  and  approximate  fuzzy  inference,  so  that  necessary 
system  performance  and  stability  can  be  achieved.  Figure  2-2  shows  a  standard  structure  of 
fuzzy  control  system. 


2.3  Why  fuzzy  sliding  control 


Sliding  mode  control  is  a  form  of  nonlinear  control,  which  is  robust  and  global  stable.  A 
sliding  controller  design  provides  a  systematic  approach  to  the  problem  of  maintaining 
stability  and  consistent  performance  in  the  face  of  modeling  imprecisions.  This  control 
design  is  conducted  in  the  hyperplane  of  the  chosen  variable  space.  In  application,  a 
boundary  layer  enclosing  the  switching  hyperplane  is  needed  to  eliminate  the  chattering 
phenomenon  [Slotine,  1991].  In  this  thesis,  the  concept  of  sliding  control  is  extended  to 
fuzzy  set  theory  for  the  purpose  of  achieving  a  new  control  which  can  have  good  robustness 
and  also  is  able  to  deal  with  unmodeled  human  expertise. 

Kung  [1994],  Ghalia  [1995]  and  Palm  [1992]  presented  the  idea  of  using  fuzzy  logic  to 
solve  the  chattering  problem  for  sliding  mode  control  and  further  developed  the  fuzzified 
sliding  mode  control,  in  which  the  switch  condition  has  been  fuzzified.  To  develop  stable 
fuzzy  control,  other  approaches  have  been  proposed  based  on  Lyapnov  stability  theory 
[Langari,  1990,  Tanaka,  1992].  By  observing  that  both  the  fuzzy  controller  and  sliding 
mode  control  can  be  visualized  and  designed  in  the  phase  plane,  Hu  [1994, 1995]  developed 
another  design  approach,  which  is  based  on  the  rule  base  of  the  human  expert.  Using  the 
fuzzified  sliding  mode  control  law  the  control  rules  can  be  tuned  in  the  phase  plane.  In  this 
thesis,  the  proposed  design  methodology  explicates  fuzzy  partition,  rule  tuning,  and 
dynamic  adjusting  of  fuzzy  partitions.  Making  use  of  the  inherent  advantages  of  sliding 
mode  control,  fuzzy  sliding  control  can  have  robustness  over  model  uncertainties  and  also 
have  the  capability  of  rejecting  disturbances. 

Considering  the  uncertainties  from  the  task  environment  and  the  human  operator,  fuzzy 
sliding  control  has  been  shown  to  overcome  these  difficulties  because  it  can  supply  the 
robots  with  varying  stiffness  control  when  the  slave  robot  contacts  the  task  environment.  A 
special  control  structure  associated  with  FSC  was  also  designed  to  deal  with  a  time  delay.  In 
the  following  sections,  the  FSC  design  methodology  and  the  control  scheme  for  a 
telerobotic  system  are  described  in  detail. 


2.4  Theory  of  Fuzzy  Sliding  Control 

In  studies  of  fuzzy  sliding  control,  there  are  two  different  approaches.  One  approach  is  to 
directly  fuzzify  the  switching  variable  S  (a  fuzzified  sliding  mode  control).  According  to  the 
extension  principle  of  fuzzy  sets  [Zadeh,  1965],  any  conventional  continuous  controller  can 
be  extended  into  a  fuzzified  controller  through  fuzzifying  appropriate  variables.  Fuzzified 


sliding  mode  control  can  attenuate  the  chattering  by  introducing  the  fuzzy  variable  of  S 
[Ghalia,  1995],  which  can  be  applied  to  a  high  order  system.  In  the  second  approach,  the 
fuzzy  sliding  control  law  is  derived  and  used  to  re-design  or  tune  the  fuzzy  logic  rule  base 
so  as  to  achieve  a  stable,  robust  fuzzy  controller.  In  this  paper,  we  focus  on  the  second 
FSC  approach. 

Let’s  consider  a  2nd  order  system  and  develop  the  corresponding  fuzzy  sliding  mode 


control  law  in  the  phase  plane. 

x(t)  =  f(x,t)  +  u(t)  +  d(t)  (2.1) 

where  x(t)  is  the  state  variable,  u(t)  is  a  control  variable,  d(t)  is  a  disturbance.  Assume 
xd(t)  is  the  desired  state,  and  the  tracking  error  is 

e  =  x(t)-xd(t)  (2.2) 

The  sliding  surface  (or  switch  line)  is:  s(x,  t)  =  0 

where  s(x,  t)  =  e+  A  •  e  A  >  0  (2.3) 

By  choosing  the  Lyapunov  function, 

V(f)  =  ±s(x,t)2  (2.4) 

the  reaching  condition  of  sliding  mode  control  can  be  written  in  the  form  of 

V  =  s- s  =  s(Xe  +  e)<-rj\s\<0 .  r}  >  0  (2.5) 

Then  from  the  reaching  condition  and  the  above  equations,  we  can  derive  the  sliding  mode 
control  law  as  follows. 

The  1st  form  of  the  sliding  control  law  is: 

u  =  -K  ■  sgn(s)  (2.6) 

where  K>rj  +  [f(x,  t)  +  d  +  Xe-xd  ]sgn($)  (2.7) 


It  is  assumed  that  the  upper  bounds  exist,  |Ae| <E,  \f(x,  f)|  <  F,  |d(0|  - 


\xd(l)\  <  v. 

K  can  be  selected  as  K=  T]  +  F+D+E  +  v, 

The  2nd  form  of  the  sliding  control  law  is:  u  =  -  Xe  -  K*  ■  sgn(s)  (2.8) 

where  K*  >  T]  +  [f(x,  t )  +  d(t )  -  x<i]5gn(5)  (2.9) 

Select  K*  as  K*  =  r]  +  F+D  +  v  (2.10) 

1  ifx  >  0 

sgn(x)  =  •  0  ifx  =  0  (2.11) 

-1  ifx  <  0 
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To  attenuate  the  chattering  in  the  neighborhood  of  the  switching  line,  a  boundary  layer  is 
introduced  [Slotine,  1991].  Based  on  (6),  we  can  obtain  the  fuzzy  sliding  control  law: 

U  =  ~Kfuzz  ■  <2-12) 

where  (s)  is  a  nonlinear  function  of  ls(t)l. 


Figure  2-3:  Fuzzified  control  value 

Here  ls(t)l  indicates  the  distance  from  a  system  state  (e,e)  to  the  switching  line.  Based  on 
the  above  fuzzy  sliding  control  law  (12),  the  fuzzy  partition  of  control  output  can  be 
determined.  The  FSC  law  can  be  interpreted  as  the  following  FSC  rules  in  the  scaled  phase 
plane. 

Rule  1:  IF  the  state  (e,  e)  is  above  the  switching  line,  THEN  u  is  negative. 

Rule  2:  IF  the  state  (e,  e)  is  below  the  switching  line,  THEN  u  is  positive. 

Rule  3:  IF  ls(t)l  is  Z  (Zero),  THEN  u  is  Z . 

Rule  4:  IF  ls(t)l  is  S  (Small),  THEN  u  is  S. 

Rule  5:  IF  ls(t)l  is  M  (Middle),  THEN  u  is  M. 

Rule  6:  IF  ls(t)l  is  B  (Big),  THEN  u  is  B. 

Generally,  by  applying  the  above  FSC  rules  and  modifying  the  original  fuzzy  control 
rules  which  have  been  abstracted  from  human  knowledge,  a  set  of  adjusted  fuzzy  control 
rules  can  be  obtained.  The  above  rules  shape  the  scaled  phase  plane  into  soft  boundary 
layers.  Figure  2-4  illustrates  the  fuzzy  partition  and  FSC  soft  boundary  layer  in  the  scaled 
phase  plane.  The  final  version  of  FSC  rules  base  is  organized  in  the  form  of  Figure  2-5. 
The  complete  rule  sets  described  in  Figure  2-5  have  been  applied  in  the  telerobot  control 
where  e(  t)  =  6d  -  9  (refer  to  Figure  3-2). 


Fuzzy  Soft  Boundary  Layer 

A*' 


Switching  Line 


Figure  2-4:  FSC  in  scaled  phase  plane 


•* 


Figure  2-6:  Control  surface  of  FSC  in  3-D. 
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2.5  Design  of  fuzzy  sliding  control 


We  summarize  our  FSC  controller  design  procedure:  1)  set  up  the  preliminary  control  rules 
and  fuzzy  partitions  from  expert  knowledge;  2)  choose  the  switching  line  and  the  soft 
boundary  layer;  3)  tune  the  fuzzy  partitions  (membership  functions)  of  the  control  variable 
and  those  for  the  input  variables  and  control  variable;  4)  modify  the  control  rules  according 
to  the  FSC  law;  5)  implement  the  fuzzy  sliding  control  on  the  general  fuzzy  control  system 
structure;  6)  simulate  and  evaluate  the  control  system,  reiterate  from  step  2  until  satisfactory 
responses  are  obtained. 

Considering  the  robot  control  applications,  it  is  necessary  to  tune  the  scaling  factors  and 
the  fuzzy  membership  functions,  i.e.  soft  boundaiy  layer  tuning,  which  is  described  in  next 
section. 


Chapter  3 

Fuzzy  Sliding  Control  of  Telerobotic  Systems 

3.1  System  structure  of  fuzzy  sliding  control 

Stable  force  reflecting  control  is  very  important  for  any  teleoperation  system  with  a  large 
time  delay.  For  some  particular  cases,  like  a  telesurgery  systems,  precise  trajectory  control 
is  also  required.  The  passive  compensation  approach[Niemeyer,  1991]  aims  to  modify  the 
communication  link  into  a  passive  transmission  link,  and  hence  smooth  the  force  signal  and 
dissipate  the  energy  generated  in  the  human  operation  loop.  But  it  has  been  found  that  the 
passive  compensation  approach  has  a  position  drifting  problem[Niemeyer,  1992].  In  tests 
of  teleoperation,  it  has  been  found  that  unstable  force  reflecting  takes  place  whenever  a 
discontinuous  force  signal  transmitted  from  the  slave  robot  is  present  at  the  master  robot 
site  and  exerts  on  the  human  operator’s  hand,  which  in  turn  produces  a  force  impulse  and 
charges  energy  into  the  operation  loop.  In  our  approach,  we  design  a  FSC,  that  dissipates 
the  energy  locally  at  the  master  robot  site  as  well  as  the  slave  robot  site,  and  stabilizes  the 
system.  The  following  system  design  can  achieve  stable  force  reflecting  and  accurate 
trajectory  control. 

In  teleoperation,  three  different  operation  modes  can  be  identified:  free  motion  mode, 
transition  mode,  and  contact  mode.  The  transition  mode  is  the  mode  in  which  an  operation 
changes  from  a  free  motion  mode  (without  environment  constraint)  into  a  contact  mode 
(with  environment  constraint)  or  vice  versa.  For  different  operation  modes,  the  controller 
may  need  to  change  for  better  performance.  A  master-slave  telerobotic  system  usually 
consists  of  two  identical  robot  arms.  Since  the  dynamics  of  each  d.o.f.  is  decoupled  from 
others,  we  only  need  to  control  each  degree  of  freedom  separately.  Therefore  we  only 
discuss  an  one-d.o.f.  control  system  in  this  paper. 

The  overall  control  system  structure  is  described  in  Figure  3-1.  In  the  telerobotic  system, 
there  are  two  robot  controllers,  i.e.  a  slave  controller  and  a  master  controller.  They  are 
hybrid  controllers  consisting  of  FSC  and  force  control  (in  Figure  3-2).  There  are  two 
supervisors,  i.e.  a  slave  supervisor  and  a  master  supervisor,  which  are  fuzzy  inference 
engines  for  changing  operation  modes  and  for  tuning  controller  parameters.  The  time  delay 
indicates  the  communication  (or  data  transmission)  time  delay  between  the  master  robot  and 
the  remote  slave  robot. 
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Figure  3-1:  Control  system  structure  of  a  telerobotic  system 

3.1.1  Controllers  in  slave  robot  and  master  robot. 

The  controller  structures  of  the  slave  robot  and  master  robot  are  the  same  as  Figure  3-2.  In 
Figure  3-2,  0d  is  the  desired  position,  and  Fj  is  the  desired  force  signal.  Kp,  IQ  are  the 
scaling  parameters  of  FSC.  There  is  an  operation  mode  parameter  a  in  the  control  system. 
When  a=0,  the  controller  becomes  a  pure  force  controller,  and  when  a=l,  the  controller  is 
a  pure  FSC  position  controller.  If  0<a<l,  the  controller  is  a  hybrid  position  and  force 
controller.  In  addition,  there  is  an  adjustable  parameter  d  of  FSC,  which  is  the  fuzzy  dead 
zone  width.  Tuning  the  parameters  a,  Kp,  IQ,  and  d  depends  on  the  operation  mode, 
which  is  governed  by  the  two  fuzzy  supervisors. 

For  the  control  of  the  slave  robot,  the  controller  is  designed  as  a  combination  of  position 
control  and  force  control.  In  the  contact  mode,  force  control  will  function,  but  in  the 
transition  mode  and  the  free  motion  mode  the  FSC  position  will  be  dominant  in  controlling 
the  robot.  In  the  transition  mode,  the  FSC  functions  as  a  variable  stiffness  controller  within 
the  soft  boundary  layer.  The  supervisor  observes  the  operation  status  of  the  slave  robot, 
estimates  the  contact  situation,  and  switches  the  control  modes  by  means  of  tuning  the 
coefficient  a  (0<a<l).  Here  the  supervisor  is  a  fuzzy  inference  engine.  Its  inputs  are  the 

computed  force  of  the  task  environment  and  the  position  responses,  and  the  outputs  are  the 
parameter  a  and  the  scaling  parameters,  Kp,  IQ  . 

In  the  master  control  subsystem,  Fd  is  the  delayed  force  transmitted  from  the  slave  robot. 
And  the  supervisor  estimates  the  slave  operator  status,  tunes  the  parameter  a,  and  the  FSC 
parameters  (i.e.  scaling  parameters  Kp,  IQ  and  fuzzy  dead  zone  width  d). 
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Figure  3-2:  Master/Slave  Controller 

3.1.2  Fuzzy  Supervisors 

•  Slave  fuzzy  supervisor 

By  observing  the  operation  status  of  the  slave  robot,  the  supervisor  can  change 
controllers  in  different  operation  modes  by  a.  Also  it  does  the  tuning  of  the  scaling 
parameters  Kp,  Kd  for  the  precise  trajectory. 

•  Master  fuzzy  supervisor 

Through  estimating  the  operation  of  the  slave  robot,  the  master  supervisor  changes  the 
control  modes  by  a.  In  the  free  motion  mode  and  contact  mode,  the  master  control  is  a 
hybrid  control  (FSC  position  control  and  force  control).  In  the  transition  mode  the  master 
control  becomes  a  FSC  position  controller,  and  the  supervisor  tunes  the  parameter  a,  the 
scaling  parameters  Kp,  Kd  and  the  dead  zone  width  d. 

•  Tuning  the  scaling  parameters 

Rule  1:  IF  the  operation  is  stabilized,  THEN  Kp,  Kd  increase  (slave  robot  only). 

Rule  2:  IF  the  operation  is  in  transition  mode,  THEN  Kp,  Kd  decrease. 

Rule  3:  IF  the  operation  is  in  free  motion  mode,  THEN  Kp,  Kd  are  kept  constant. 

Among  the  above  rules,  Rule  1  is  applied  when  the  state  (e,  e)  stays  within  the  soft 
boundary  layer  and  there  is  no  oscillation  in  the  operation.  Rule  1  changes  the  boundary 
layer’s  shape  as  shown  in  Figure  3-3,  which  gives  better  tracking  accuracy  than 
conventional  sliding  control.  In  transition  mode,  decreasing  Kp,  Kd  means  increasing  the 
width  of  soft  boundary  layer  and  thus  adjusts  the  robot  stiffness  relatively  in  a  physical 
sense. 

•  Tuning  the  dead  zone  width  d  (master  robot  only) 

Rule  4:  IF  a  discontinuous  force  feedback  appears,  THEN  increase  the  dead  zone 
width. 

Rule  5:  IF  the  position  response  is  disturbed,  THEN  increase  the  dead  zone  width. 


Rule  6:  IF  the  position  response  is  smooth,  THEN  the  dead  zone  width  is  zero. 

In  the  master  supervisor  module,  Rule  4,  Rule  5  and  Rule  6  are  called  fuzzy  force  filter 
which  will  shield  the  disturbances  and  the  jumping  force  signals,  and  can  help  to  stabilize 
the  force  reflecting.  Figure  3-4  shows  the  tuning  of  the  fuzzy  dead  zone. 


▲  e' 


e 


Figure  3-3:  Tuning  the  fuzzy  membership  function  by  the  scaling  factors 
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Figure  3-4  Tuning  the  fuzzy  dead  zone  width 


3.2  Modelling  of  the  manipulator,  human  arm  and  environment 

Precisely  modeling  a  teleoperation  system  is  not  a  easy  job  since  the  uncertainties  of  the  task 
environment,  human  arm  and  the  dynamics  of  the  robot  arms  exist  in  the  subsystems.  A 
teleoperation  system  contains  two  manipulators;  a  master  which  only  interacts  with  a  human 
operator,  and  a  slave  which  only  interacts  with  a  task.  There  are  no  direct  cross  interactions 
between  the  master  and  the  task,  or  between  the  slave  and  the  human  operator.  The  human  operator 
and  the  task  both  have  their  own  dynamics.  When  interacting  with  a  teleoperator  system,  both 
human  operator  and  task  have  the  potential  to  destablize  the  coupled  system.  Consequently,  the 
interactive  nature  of  the  teleoperation  system  demands  stability,  namely  so-called  coupled  stability 
when  the  teleoperation  system  is  coupled  with  both  the  human  and  the  task.  In  terms  of  stability 


analysis  and  control  design,  because  a  human  operator  can  change  his  arm  dynamics,  and  the  task 
dynamics  vary  from  task  to  task,  the  controller  of  a  teleoperator  should  be  robust  enough  to  handle 
various  human  and  task  dynamics.  In  this  section  the  structure-determinant  model  is  pursued  for 
every  subsystem  of  the  teleoperator  system. 

3.2.1.  Model  of  the  manipulator 

A  Teleoperation  system  is  different  from  a  traditional  manipulator.  Usually  a  traditional 
manipulator  directly  interacts  with  the  task  environment  and  the  dynamics  of  its  joints  are  coupled. 
But,  for  a  master-slave  teleoperation  system,  the  dynamics  of  different  joints  are  decoupled,  thus 
the  control  design  can  be  carried  out  joint  by  joint  (one  d.o.f  at  a  time).  The  robots  that  we  use  are 
Phantom  robot  arms.  The  dynamic  uncertainty  of  each  joint  can  be  ignored  because  of  its  low 
inertia  structure.  Therefore,  the  master-slave  manipulator  system  can  be  modeled  as  a  one  d.o.f. 
motor  driven  subsystem.  Figure  3-5  shows  the  block  diagram  of  the  one  d.o.f.  control  system. 
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3.2.2.  Model  of  human  arm 

Since  the  human  operator  is  inherently  part  of  a  teleoperation  system,  the  system’s 
dynamics  can  not  be  completely  described  without  taking  the  operator’s  arm  into 
consideration.  However,  due  to  the  complicated  dynamics  of  the  human  arm,  an  accurate 
model  of  arm  impedance  is  not  yet  available  in  the  literature.  For  most  analyses  of 
teleoperation,  a  human  operator  model  has  not  been  included.  Particularly,  in  passive 
control  approaches,  people  always  assume  that  the  operator  arm  can  be  treated  as  a  passive 
impedance  model.  In  fact,  this  is  not  true.  One  should  consider  the  effect  of  arm 
locomotion  during  teleoperation.  Because  of  arm  locomotion,  an  operator  can  generate  a 
force  and  position  impulses  in  reacting  to  a  step  force  signal  and  the  operator  can  become 
fatigued.  Hannaford  and  Anderson  [1988]  applied  a  linearized  version  of  a  sixth  order  non¬ 
linear  model  of  one  axis  human  movement  in  modeling  their  teleoperation  system.  The 
operator  was  shown  to  have  a  significant  effect  on  the  dynamics  of  the  system.  Another 
example  can  be  found  in  the  work  done  by  Lee  and  Lee  [1992].  They  derived  a  third  order 
model  of  the  operator  from  considerations  of  the  intentional  mechanisms  in  the  brain,  the 
neuromuscular  response  and  the  human  operator’s  arm  dynamics.  It  has  been  shown  by 
Lee  and  Lee  that  a  system,  without  human  dynamics  and  stable  under  time  delay,  can 
become  unstable  when  a  simplified  model  of  the  operator  dynamics  is  included. 

Chin  [1991]  summarized  several  different  models  from  a  systematic  and  physiology 
point  of  view,  in  which  the  linear  lumped  parameter  muscle  model  and  arm  locomotion  are 
specified.  Figure  3-6  is  a  second  order  model  of  a  human  arm. 


Kp 


Figure  3-6:  A  model  for  a  human  arm  (a  lumped-parameter  muscle  model).  B  is  a  damping 
parameter,  K  is  a  elastic  parameter  for  the  human  arm.  T  is  the  arm  power  source,  i.e.  arm 
locomotion. 

3.2.3  Model  of  task  environment 

The  concepts  of  impedance  and  admittance  are  useful  in  considering  the  interaction  of  the 
slave  with  the  remote  task  and  the  interaction  of  the  human  operator  with  the  master. 
Impedances  accept  flow  (e.g.  velocity)  and  produce  effort  (e.g.  force)  whereas  admittances 
accept  effort  and  produce  a  flow.  As  Hogan  [1985]  observes,  whenever  a  dynamic 
interaction  between  two  physical  systems  is  analyzed  the  representations  must  complement 
each  other.  Thus,  if  the  environment  is  represented  as  an  impedance,  it  accepts  a  velocity 
command  and  produces  a  force  as  an  output.  Based  on  this  consideration,  Niemeyer  and 
Slotine  [1991]  proposed  to  use  impedance  matching  terminals  in  their  passive  control 
model. 

In  the  research  of  telesurgery,  our  task  environment  is  the  patient  body,  particularly  the 
abdomen  of  the  patient  in  laparoscopic  surgery.  To  simplify  the  physical  environment  a 
simple  task  model  simulating  the  telemanipulator  interaction  with  the  tissue  of  the  patient’s 
body  was  provided  by  a  mechanical  beam.  Normal  force  applied  is  nonlinear  with  both 
tangential  position  but  linear  with  the  normal  displacement,  much  as  would  be  encountered 


in  surgery.  This  provided  a  good  test  of  our  methods  for  coping  with  time  delay.  Figure  3- 
7  shows  the  task  model. 


Figure  3-7:  A  task  model  simplified  for  telesurgery 


3.3  Stability  of  the  force  reflecting  and  the  trajectory  accuracy 

To  prove  the  overall  system  stability  of  a  teleoperation  system  with  FSC,  which  includes 
human  operator  in  the  loop,  one  should  consider  the  human  arm  dynamic  model  as  well  as 
human  mental  control  model.  The  following  is  the  structure  of  a  teleoperation  system 
proposed  in  the  thesis  (Figure  3-8). 

In  this  system,  the  models  of  different  system  components  are  explained  as  follows: 

e~sT :  telecommunication  time  delay  (T  seconds); 

Gs(s):  slave  robot  model  (2nd  order) ; 

Gs c  ( s ) :  model  of  slave  controller; 

Ggv(s):  task  environment  model  at  the  slave  site; 

Gm  ( s ) :  master  robot  model  (2nd  order) 

Ghm(s):  model  of  a  human  arm; 

Ghc(s):  model  of  human  mental  control,  which  can  be  treated  as  a  PD 

controller  combined  with  a  time  delay  about  0.1  sec.[Lee  and  Lee,  1992]. 

The  model  of  fuzzy  filter  (a  part  of  FSC  in  the  telerobotic  system)  can  be  treated  as  a  low 
pass  filter  in  case  of  a  step  signal  input  in  a  mathematic  sense,  but  functionally  it  is  the  logic 
combination  of  a  class  of  low  pass  filters. 


The  stability  proof  can  be  accomplished  by  means  of  the  corollary  of  the  small  gain 
theorem  described  in  Appendix  A.  Let’s  start  from  the  most  inner  loop,  loop-1.  In  loop-1, 
since  the  FSC  is  applied,  the  subsystem  in  this  loop  behaves  as  a  1st  order  passive  system 
as  long  as  the  system  states  reaches  the  switching  surface  (or  switching  line).  Hence  the 
transfer  function  of  loop- 1  is  strictly  small  gain.  In  loop-2,  the  subsystem  can  be  simplified 
into  the  form  of  Figure  A-l.  By  applying  the  corollary  in  appendix  A,  we  know  that  the 
loop-2  is  stable  and  the  transfer  function  in  loop-2  is  small  gain  too.  Similarly,  in  loop-3, 
we  can  also  prove  the  stability  by  the  corollary  of  the  small  gain  theorem.  Therefore,  the 
overall  system  is  stable.  The  more  detailed  mathematic  proof  will  be  found  in  the  future 
publications. 

The  philosophy  of  the  stable  force  reflecting  control  in  this  system  structure  is  to 
dissipate  the  energy  in  the  master  robot  control  loop,  which  is  different  from  the  passive 
compensation  approach  [Anderson  and  Spong,  1989].  For  the  master  robot,  we  can  model 
the  human  arm  as  a  passive  impedance  structure  plus  a  particular  power  source  (impulse 
effort  source).  The  goal  of  the  FSC  design  is  to  choose  the  appropriate  soft  boundary  layer 
and  the  nonlinear  control  output  u  within  the  soft  boundary  layer,  and  also  to  select  the 
dead  zone  parameter  d  appropriately  such  that  the  local  control  loops  at  the  master  site  and 
the  slave  site  can  satisfy  the  Small  Gain  Theorem  respectively.  If  the  small  gain  theorem 
can  be  satisfied  in  both  local  control  loops  (master/slave  control  loop),  the  local  stability  is 
guaranteed  and  thus  the  stability  of  the  telerobotic  system  is  achieved. 

The  trajectory  accuracy  of  the  slave  robot  can  be  obtained  by  means  of  the  FSC  in  the 
slave  controller.  The  soft  boundary  layer  in  the  FSC  can  stabilize  the  tracking  in  the 
transition  mode,  and  by  tuning  of  the  scaling  parameters  Kp,  K<j  the  trajectory  accuracy  can 
be  improved  (refer  to  Figure  3-3). 
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Figure  3-8:  A  complete  system  diagram  of  teleoperation 


Chapter  4 

Other  Control  Approaches 

4.1  Raju’s  General  Passive  Control 

Raju  [1986,1989]  first  modeled  a  master-slave  manipulator  system  into  a  two-port  network 
and  applied  the  two-port  electric  network  theory  to  explore  the  stable  force  reflection 
control  for  the  teleoperation  system.  Figure  4-1  shows  Raju’s  two-port  network  model  for 
master-slave  teleoperation  system. 


Operator 


Task 


Figure  4-1:  Raju’s  two-port  network  model 


Z  represents  compliance-viscosity-inertance  (impedance),  T  represents  joint  torque,  and 
Q  represents  joint  velocity  (for  master  m  or  slaves),  all  in  frequency  domain.  Zh  and  Zt 

are  the  given  impedance  characteristics  of  the  human  neuromuscular  system  and  of  the  task, 
respectively,  and  Zm  and  Zs  are  port  equivalent  impedances  looking  into  the  master  side 

and  into  the  slave  side  respectively  of  the  master-slave  manipulator.  From  circuit  theory, 
we  define  the  following, 


Therefore, 

Z.=TJam=Z >1-[Z,2Z2,AZ22  +  Z,] 

Z,  =  TJas  =  Z22  -  [Z12Z2l/(Zll  +  Z*] 
Let  the  control  law  for  master  and  slave  as. 


(4.1) 

(4.2) 

(4.3) 


UJs)  =  -(^i  +  Kl2)Qm(s )  +  (i.  +  *14)Q,(s)  (4.4) 

5  S 

U,U)  =  -(^  +  Kn)Qm(s)  +  fe  +  ^>£2,(5)  (4  5) 

5  s 


Then  by  superposition  (assuming  linearity  here) 
T=  Z  Qm  -  U 

m  ma  m  m 

ts  =  za  -  (/ 

*  5<Z  J  5 


(4.6) 

(4.7) 


where  Z^  and  Zsa  are  equivalent  impedances  looking  into  master  and  slave  arms  without 

the  influence  of  feedback  control  on  the  respective  actuators.  Then  from  the  above 
equations  we  can  obtain  Zu,  ZJ2,  Z2V  Z22,  .  Raju  showed  that  by  choosing  Ktj 

appropriately,  one  can  ensure  that  the  master-slave  manipulator  system  is  passive,  namely 
Zjj  is  strictly  positive,  thus  stable  for  any  passive  termination  Zh  at  the  master  port  and 

any  passive  termination  Z(  at  the  slave  port.  From  all  his  experiments  Raju  concluded  that 
adjustable  impedance  is  desirable  when  a  master-slave  manipulator  is  to  be  used  in  tasks 
made  up  of  subtasks  with  different  characteristics. 

This  approach  is  a  general  framework  for  passive  control,  and  a  lot  of  research  has  been 
based  on  it  afterwards.  However  there  is  no  guideline  to  tell  people  how  to  choose  the 
control  parameters  such  that  the  optimal  passive  control  can  be  achieved.  Chin  [1991] 
further  explored  the  two-port  network  approach  and  derived  an  optimal  passive  control 
design  framework.  He  obtained  the  necessary  and  sufficient  conditions  for  stable  passive 
control. 


4.2  Scattering  Theory  and  Conservative  Passive  Control 

4.2.1  Scattering  Theory 

Inspired  by  Raju’s  two-port  network  model,  Anderson  and  Spong  [1989],  presented  a 
method  based  on  merging  velocity  and  force  signals,  called  scattering  theory,  which  relies 
on  the  passivity  of  the  master  and  the  slave  systems  connected  to  each  side  of  the 
communication  channel.  Niemeyer  and  Slotine  developed  a  similar  algorithm,  but  included 
terminators  at  both  ends  of  the  communication  line.  Using  the  analogy  between  electrical 
and  mechanical  systems,  the  teleoperator  system  can  be  modeled  as  a  network  of  two-port 
elements  representing  the  master,  communication  link  and  the  slave  subsystems.  At  both 
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ends,  two  one-port  elements  complete  the  bilateral  control  scheme:  the  operator  and  the 
environment,  see  Figure  4-2.  In  general,  the  human  operator  commands  a  position  (or  a 
velocity)  to  the  master  subsystem  and  receives  back  information  about  the  interaction  forces 
exerted  on  the  slave  by  the  environment. 


Figure  4-2:  A  2-port  network  transmission  line  model  for  a  telerobotic  system 


A  hybrid  matrix  (or  scattering  operator)  representation  has  been  widely  used  to  describe 
the  performance  of  a  teleoperation  system.  The  description  of  the  teleoperator  is  based  on 
the  relationship  between  effort  and  flow  in  each  two-port  [Raju,1986]. 

The  hybrid  matrix  (or  scattering  operator)  can  be  defined  as, 


fM 

-xs(s) 


=  H(S) 


*mW 

fs(s) 


(4.8) 


The  Scattering  Theorem  states  that  a  two  port  network  is  passive  if  and  only  if  the  norm 
of  its  scattering  operator  S  is  less  than  1,  where  S  can  be  defined  for  a  two-port  network  by 
the  relationship  between  the  effort  (force)  and  flow  (velocity). 

F  -  v  -  S(F  +  v)  (4.9) 

where  F  =  [/m(s)  fs(s)],  v  =  [^(s)  *5(s)] 


S  = 


1 

0 


(. H(S)-I){H(S)  +  I)~ 1 


(4.10) 


The  scattering  theory  shows  that  the  destabilizing  effects  present  in  the  tele-operator 
based  on  standard  communication  are  caused  by  the  non-passivity  of  the  communications 
network.  To  obtain  a  passive  system  in  the  presence  of  any  time  delay,  a  lossless 
transmission  line  connection  has  to  be  considered.  From  an  energy  point  of  view,  this 
passive  element  does  not  increase  the  energy  of  the  overall  system.  In  fact  the  passivity 
condition  states  that  in  a  passive  n-port,  power  can  be  dissipated  or  stored.  The  definition 
of  hybrid  and  scattering  matrices  and  the  passivity  conditions  on  their  norms  give  tools  to 
prove  the  passivity  of  elements  in  the  teleoperator.  By  means  of  energy  considerations  and 
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definition  of  the  scattering  matrix,  lossless  transmission  lines  are  shown  to  be  passive 
(even  if  not  strictly  passive)  for  any  time-delay  of  the  communication  network.  In  this  case 
the  overall  system  is  obtained  as  a  connection  of  only  passive  elements  and  therefore  it  is 
shown  to  be  passive. 


4.2.2  The  equivalence  between  Anderson’s  approach  and  Niemeyer’s 
approach 


(1)  Anderson  and  Spong  Scheme[1989] 

In  this  scheme,  the  communication  network  is  obtained  by  emulation  a  passive  lossless 
transmission  line  and  adding  two  equivalent  transformers  at  both  ends  (B  and  1/B)  to 
overcome  problems  with  the  scaling  between  force  and  position/  velocity  domains.  The 

resulting  system  described  in  power  variables  is: 

/„,<»)  =  /,(»  -  T)  +  BUJ,)  -  xjt  -  D)  (4.11) 


*w(0  -  i.(f- r) +  ![/*(» -T) -/,<»)] 


B 


(4.12) 


Using  the  simple  dynamics  of  the  master  and  slave,  a  control  scheme  based  on  a  local 
slave  impedance  control  was  proposed  by  Anderson  [1989]. 

The  resulting  equation  of  the  master  and  slave  dynamics  and  controllers  are: 

Mm*m  =  fh  +  Tm 

MS  =  ~f  +  T 


T  =  -B' 

m 


_  X  -  f 

mm  ^  j 


md 


a  rf 

fJenv 


(4.13) 

(4.14) 

(4.15) 

(4.16) 

(4.17) 


T  =  —B'  x  —  f 

S  S  S  J  S 

f  =  K'  [A xdt  +  B'  Axs 

J  S  S  J  S  S  4 

(2)  Niemeyer  and  Slotine’s  scheme[1991] 

In  Niemeyer’s  scheme,  the  dynamics  of  the  communication  network  using  wave 
variables  is  expressed  as: 


*«,<»>---£  [/,(')  -V2®vs(»] 

where 

um(t)  =  V2 Bx(t)  +  vm(r) 


(4.18) 

(4.19) 

(4.20) 
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(4.21) 

(4.22) 

(4.23) 


v  (0  =  um(t-  T ) 
vjt)  =  us(t  -  T) 


The  modified  passive  teleoperation  communications  network  is  shown  in  Figure  4-3, 
which  is  obviously  equivalent  to  the  system  described  by  (4. 1 1, 4. 12) 

While  using  the  transmission  line  as  a  carrier  of  the  signals  in  the  tele-operator,  care 
must  be  taken  in  the  adaptation  of  both  terminals  where  the  impedance  of  the  line  changes 
and  reflections  occur.  To  avoid  the  reflections  at  the  extremities  of  the  line,  Niemeyer 
proposed  the  use  of  terminator  elements.  The  impedance  of  the  terminators  must  match  the 
characteristic  impedance  of  the  transmission  line.  Figure  4-4  represents  the  passive 
teleoperation  communication  network  modified  with  terminators  written  in  the  wave 
variables  domain. 

The  terminator  equations  are  described  by 

x'm(t)  =  xjt)-  ± /""(,)  .  (4.24) 

fs(0  =  fs(t)  +  Bxsd(t)  (4.25) 
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4.3  The  remained  problem  with  the  passive  control  approaches 


(1)  Passive  control  with  terminator  (Niemeyer’s  scheme) 

From  (4.1 1, 4.12)  and  (4.24, 4.25)  it  is  possible  to  derive  the  following: 


,  N  [W.  .  . 

Um(t)  =  ijjxjt) 

(4.26) 

“(0=V2 ~Bf‘(t) 

(4.27) 

=  f  xm^  +  \z~lfs(z) 

(4.28) 

Xsd^  =  ^~lXm(z)-^fs(z) 

(4.29) 

Where  z  =  esT  time  delay  element. 

From  (4.28)  and  (4.29),  we  can  see  the  loss  of  energy  due  to  the  presence  of  the 
terminators  makes  the  system  stable  (in  the  Z  domain).  Also  we  can  see  that  no  reflection 
occurs  at  both  ends  of  the  network  from  (4.26),  (4.27).  But  the  position  drift  problems  are 
present  since  in  steady  state  (z_1  =1  ), 

x  =-x  —  f 

xsd  2  *m  ~  2  B  Js  (4.30) 

Therefore  a  constant  value  of  f  in  the  velocity  signal  is  integrated  resulting  in  a  position 
ramp  for  the  desired  position  of  the  slave. 

(2)  Passive  control  without  terminator  (Anderson’s  scheme). 

The  scheme  proposed  by  Anderson,  (4.11)  ~  (4.17),  includes  local  controllers  at  the 
master  and  slave  sides.  Let  the  elements  of  B' m  and  B' s  be  equal  to  B.  From  (4.13)  ~ 

(4.19),  using  the  equations  of  the  controllers,  we  obtain: 


um(t)  =  JlBxm(t)  +  vm(f)  (4.31) 

=  ihi  f,(,)  -  (4.32) 

fmi (O  =  2BxJz)  +  z~xfs{z)  -  Bz~lxs(z)  (4.33) 

+  z-'^iz)  -  Bz~xxs{z)  (4.34) 


From  equation  (4),  we  see  that  the  reflection  does  exist  at  the  master  side,  but  there  is  no 
reflection  at  the  slave  side,  which  makes  the  system  stable.  We  can  get  the  same  result  from 
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(15)  in  Z  domain.  In  addition,  at  steady  state  from  (15),  we  obtain  f ^  =  fs  and 
*  =  x  Therefore,  no  position  drift  is  produced  by  the  coordinating  force  signal  in  this 

sd  m 

case. 

4.4  Modification  of  passive  control  to  compensate  position  drift. 


As  described  in  the  previous  sections,  the  presence  of  the  matching  terminators  modifies 
the  overall  communication  system  in  terms  of  power  reflections  causing  position  drift  on 
the  slave  desired  position  signals.  In  fact,  the  term  fin  (4.29)  at  steady  state  provides 

constant  input  for  the  slave  desired  velocity  and  therefore  an  increasing  position  set-point 
for  the  slave  controller.  This  effect  can  be  neglected  if  the  time  length  of  the  teleoperation 
task  is  not  considerably  large,  but  in  practical  applications  the  importance  of  the  drift 


compensation  is  evident. 

The  proposed  scheme  is  essentially  based  on  a  compliance  controller  at  the  slave  side 
[Kim,  1992].  In  place  of  the  velocity  signals  this  scheme  is  based  on  transmitting  the 
increments  of  the  sensor  position.  Each  control  cycle  the  communication  network  receives 
the  increment  of  the  master  position  and  provides  to  the  slave  side  the  increment  of  the 
desired  position.  The  control  algorithm  corrects  for  the  fs  steady  state  value  avoiding  the 

drift  problem.  This  command  is  then  integrated  to  obtain  the  set-point  for  the  slave 
position.  The  incremental  desired  position  provided  to  the  slave  compliance  controller  is 


therefore  expressed  as: 

1  . 

xsd  =  2X" 


(4.35) 


Figure  4-5  is  the  block  diagram  of  the  compensation  scheme,  which  implies  the 
insertion  of  a  block  1/2  B  at  the  slave  side.  The  value  fs  is  therefore  multiplied  by  the 
factor  1/2  B  and  then  added  to  the  original  to  compensate  for  the  steady-state 


component  of  the  coordinating  force.  The  derived  signal  is  then  used  as  input  for  the 
compliance  controller  realized  at  the  slave  side. 


sd 


+  sd 

i 


B 


2  B 


fs  +T+  j 

<4 - - L 


/. 


Figure  4-5:  An  additional  link  for  drift  compensation 
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4.5  Supervisory  control  for  optimal  performance 

As  described  in  the  previous  sections,  the  drawback  of  an  ideal  “transparent”  force 
reflection  is  the  limit  of  human  capability  in  handling  certain  tasks.  In  order  to  improve  the 
performance  of  force-reflection  teleoperation,  we  can  categorize  different  tasks  by  different 
requirements  of  the  sensory  information  and  actuation  needed  to  accomplish  the  tasks.  A 
human  operator  can  select  the  corresponding  control  models  with  respect  to  different  tasks. 
On  the  other  hand,  teleoperation  is  mostly  conducted  in  unstructured  environments  that  are 
either  hazardous  or  too  costly  for  human  operators  to  reach  directly.  There  are  no 
mathematical  models  to  describe  the  teleoperation  processes  under  different  tasks  for  the 
purpose  of  automatic  decision  making.  But  the  human  operator  can  take  care  of  it.  Through 
dynamic  masking  by  the  mechanical  structure  of  master  and  slave  manipulators,  a  human 
operator  can  still  accomplish  the  task  by  functioning  as  a  robust  and  adaptive  controller. 
With  a  large  amount  of  knowledge  accumulated  from  experience,  a  human  operator  can 
retrieve  and  modify  his  knowledge  easily  in  coping  with  new  tasks.  Consequently,  the 
human  operator  is  very  adaptive  when  there  is  little  prior  information  about  the  task  model 
and/or  when  there  is  too  much  noisy  sensory  information.  The  robust  and  fast 
computational  ability  also  allows  him  to  plan  motion  effortlessly  when  the  task  model  is 
available  but  not  accurate.  Therefore,  an  ideal  teleoperation  system  which  provides  a 
transparent  interface  between  the  operator  and  the  task  can  take  full  advantage  of  human 
capability. 

Supervisory  control  can  offer  not  only  the  teleoperation  system  with  the  capability  of 
intervening  at  various  levels  to  perform  control  along  a  continuum  from  direct  manual 
control  to  quite  high-level  supervisory  command  and  control,  but  also  the  capability  of 
coping  with  time  delay  as  well.  The  theory  of  supervisory  control  was  initially  developed  in 
the  Human  Machine  Systems  Lab  at  MIT  in  the  1960s  and  1970s.  Sheridan  [1983] 
discussed  the  concept  of  supervisory  control  extensively  in  1983.  The  advantages  of 
supervisory  control  in  teleoperation  are  the  following: 

•  Supervisory  control  encompasses  a  wide  range  of  options  for  remote  manipulation.  A 
human  supervisor  is  encouraged  to  select  the  most  effective  means  to  perform  a  certain 
task. 

•  Supervisory  control  can  avoid  the  time  delay  problem  by  closing  the  control  loop 
within  the  remote  site. 

Under  supervisory  control  (Figure  4-6)  the  operator  sends  coded  instructions  to  the 
remote  telerobot  specifying  subtask  conditions  and  procedures  with  reference  to  sensor 
states  that  must  be  satisfied.  The  operator’s  inputs  could  range  from  a  purely  manual  analog 
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command,  demonstration  in  time  and  space  what  is  intended,  to  a  highly  abstract  symbolic 
command  made  up  of  alphanumeric  keystrokes.  The  instructions  are  sent  open  loop  relative 
to  the  human.  The  remote  computer  then  interprets  the  message  and  acts  on  the  sensor 
information  available  to  it  about  its  own  environment.  The  control  loop  for  the  dynamic 
behavior  of  the  robot  is  closed  within  the  remote  site,  where  there  is  no  time  delay  in  the 
loop,  and  the  control  mode  is  selected  based  on  the  supervisor’s  command.  The  remote 
computer  then  relays  that  information  which  is  deemed  important  and  necessary  for 
effective  supervision  back  to  the  operator,  the  responsibility  for  the  specific  details  of 
control  being  left  to  the  subordinate  computer. 

Supervisory  control  of  teleoperation  has  been  investigated  for  our  project  (telesurgery). 
Some  studies  were  done  on  the  master-virtual  slave  and  virtual  task  environment.  For 
complex  teleoperation  tasks  like  telesurgery,  there  are  currently  difficulties  in  the  human- 
machine  communication  interface  on  the  supervisory  side,  which  prevents  efficient 
implementation  of  this  control  approach.  In  the  near  future,  we  believe,  with  better  virtual 
reality  technology,  supervisory  control  will  show  its  potential  in  advanced  Teleoperation, 
especially  in  telesurgery. 
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Figure  4-6:  System  structure  of  supervisory  control 
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Chapter  5 

Experimental  Studies  on  FSC  with 
a  Virtual  Environment 

In  the  previous  chapters,  the  design  methodology  of  FSC  and  the  system  configuration  of 
a  telerobotic  system  with  time  delay  were  elaborated.  It  is  interesting  to  evaluate  the  design 
of  the  FSC  controller  and  to  compare  with  some  related  controllers,  such  as  sliding  control 
with  fixed  boundary  layer,  adaptive  deadzone  control,  and  linear  PD  control  in  the  first 
stage  of  the  research.  By  taking  advantage  of  computer  graphics  and  dynamics  simulation 
techniques,  a  virtual  slave  robot  with  a  virtual  environment  has  been  developed,  which 
gives  us  convenience  to  investigate  different  control  approaches  with  respect  to  different 
time  delays.  This  chapter  addresses  the  actual  master-virtual  slave  experiments. 

5.1  Configuration  of  an  actual  master-virtual  slave  telerobotic 
system. 

In  a  scientific  sense,  performing  experiments  on  an  actual  master-virtual  slave  telerobotic 
system  with  virtual  environment  gives  the  following  advantages  in  research: 

(1)  This  system  can  supply  the  subject  with  sufficient  interaction  with  the  environment. 
The  force  reflection  is  realized  by  the  master  robot. 

(2)  The  virtual  slave  robot,  virtual  environment  and  virtual  communication  delay  box 
makes  it  possible  for  us  to  avoid  the  hardware  difficulties  and  to  concentrate  on  the 
theoretic  research  in  the  early  stage. 

(3)  With  the  virtual  environment  and  virtual  communications  delay  box,  one  can  easily 
conduct  experiments  in  different  settings  by  means  of  selecting  different 
parameters. 

(4)  Experimenting  with  the  virtual  models  reduces  the  safety  problem. 

Figure  5-1  describes  the  configuration  of  an  actual  master- virtual  slave  telerobotic 
system.  In  this  system,  there  are  the  following  components:  a  master  robot  (3  d.o.f.), 
amplifiers  box,  data  acquisition  card  (computer  interface  with  encoders,  D/A  converters ),  a 
master  robot  control  module,  slave  robot  dynamics  simulation,  slave  control,  graphics 
display,  time  delay  box  and  virtual  beam  model.  All  the  control  algorithms,  virtual  models, 
and  graphics  are  implemented  by  a  Pentium  computer.  Figure  5-2  shows  the  structure  of 
Phantom  robot  (master  robot  in  picture). 

The  mathematic  model  of  the  nonlinear  beam  model  is  described  in  appendix  C. 
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Figure  5-2:  The  structure  of  Phantom  robot 


5.2  Human  Subject  experiment  design 

In  telerobotics,  human  subject  experimenting  is  very  important  because  the  human  operator 
is  one  component  in  the  whole  telerobotic  system  and  the  human  interaction  is  not  in 
consideration  of  the  design  of  low  level  robot  controllers.  As  described  in  previous 
chapters,  during  a  teleoperation  process,  the  human  operator  is  in  the  outer  loop  of  the 
teleoperation  system  and  he/she  controls  the  teleoperation  process  through  visual  feedback 
and  force  feedback.  Improved  operation  can  be  achieved  by  means  of  a  better  human- 
machine  interface,  such  as  Supervisory  Control,  which  is  not  the  focus  of  this  research. 

The  designed  stable  controllers  are  in  the  inner  loop  of  the  master-slave  telerobotic 
system.  The  control  design  is  based  on  the  model  of  the  robot  dynamics  and  some 
assumptions  for  the  task  and  human  operator.  Evaluation  of  the  overall  system  performance 
becomes  quite  necessary  for  us  to  judge  the  system  control  design  and  verify  the 
assumption  we  made.  Therefore,  human  subject  experimenting  is  a  very  crucial  part  in  the 
design  of  a  teleoperation  system. 

Using  the  above  system  (in  section  5.10),  human  subject  experiments  can  be  easily 
conducted  for  a  particular  controller  with  respect  to  different  time  delays.  The  task  in  each 
case  was  to  move  the  real  master  robot  arm  so  that  the  virtual  slave  arm  contacted  the  virtual 
beam  at  a  given  point  from  a  free  space  starting  position,  apply  a  small  force,  then  slide 
along  the  beam  to  another  specified  point  while  maintaining  force  contact. 

Using  the  master-virtual  slave  system,  the  following  control  techniques  were  studied: 

•  Fuzzy  sliding  control; 

•  Sliding  control  with  boundary  layer; 

•  Adaptive  dead  zone  control; 

•  Linear  PD  control; 

Ten  MIT  graduate  students  were  chosen  randomly  as  subjects.  Time  delays  of  0,  100, 
500,  800,  and  1000  ms  were  tested.  During  the  experiment,  the  dynamic  response  data 
were  recorded  and  used  to  do  objective  evaluation  based  on  the  vibration  magnitude. 
Subjective  difficulty  evaluations  were  also  made  based  on  a  subjective  rating  guide. 


5.3  Results  Analysis. 

Figure  5-3  shows  the  statistics  box  plots  of  the  subjects’  experimental  data  in  four  different 
cases:  traditional  PD  control;  Fuzzy  Sliding  Control  (FSC);  Sliding  Mode  Control  with 
Boundary  Layer  (SMC/BL);  Adaptive  Dead  Zone  Control  (Adaptive  DZC).  The 


information  shown  in  the  plot  is  the  subjective  “Easiness”  rating,  which  depends  on  the 
control  stability  of  robots,  magnitude  of  position  vibration  and  the  fatigue  tolerance  of  the 
operator’s  arm. 

Plotted  in  Figure  5-4  are  the  typical  dynamic  responses  of  the  subject  teleoperation 
experiments  (1  second  time  delay).  The  measurement  of  vibration  magnitude  of  the  position 
dynamic  responses,  which  were  recorded  during  every  subject’s  experiment,  gives  us  the 
objective  evaluation  of  the  controllers  with  respect  to  the  time  delays. 

Comparisons  of  the  four  controllers  are  shown  in  Figure  5-5  and  Figure  5-6.  In  Figure 
5-5,  the  subjective  evaluations  are  based  on  the  mean  value  of  the  corresponding  subjective 
“Easiness”  rating.  Similarly  the  objective  evaluations  are  plotted  based  on  the  mean  values 
of  the  objective  measurement  of  the  position  vibrations  (Figure  5-6). 

The  Conclusions  we  can  draw  from  the  comparisons  are  the  following: 

1)  From  the  comparison  plot  of  subjective  evaluation  (Figure  5-5),  it  is  shown  clearly 
that  FSC  was  the  best,  and  PD  control  was  the  worst  control  approach  in  dealing 
with  time  delay.  SMC/BLand  Adaptive  DZC  control  approaches  were  close  to 
the  FSC  approach. 

2)  From  the  comparison  of  objective  evaluations  (Figure  5-6),  we  observe  that  the  best 
control  is  FSC,  and  PD  control  is  the  worst  case.  The  other  two  approaches, 
Adaptive  DZC,  and  SMC/BL  are  better  than  PD  and  close  to  the  FSC  approach. 

3)  While  the  time  delay  increases,  the  performance  of  the  teleoperation  degrades. 

4)  The  critical  time  delay  point  is  roughly  at  100  ms  according  to  our  experiment 
system,  which  means  that  when  time  delay  becomes  larger  than  0. 1  second  the 
system  performance  starts  to  degrade. 

The  reason  that  the  evaluation  of  SMC/BL  and  Adaptive  DZC  are  close  to  the  best 
approach  (FSC)  in  the  experiments  can  be  explained  as  follows:  FSC  is  a  special  sliding 
mode  control,  which  has  a  soft  boundary  layer  and  varying  dead  zone;  The  adaptive  DZC 
or  SMC/BL  only  has  one  of  the  characteristics,  both  of  which  are  significant  in  improving 
the  system  performance. 
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iox  plots  of  the  subject  experiment  data,  (a)  PD  control;  (b)  FSC 
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laptive  Dead  zone  control. 
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Case  a: 


Case  b: 


Figure  5-4:  Typical  dynamic  responses  from  the  subject  teleoperation  experiment  (1  second 
delay).  Case  a:  PD  control;  Case  b:  FSC  control. 
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Average  Subjective  "Easiness"  Rating 
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Average  Amplitude  of  Vibration 


Figure  5-6:  Comparison  of  controllers  by  objective  evaluations 
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Chapter  6 

Design  of  a  Telerobotic  System 

6.1  System  Configuration 

6.1.1  System  for  telesurgery  demonstration 

Figure  6-1  shows  a  diagram  of  the  telerobotic  surgery  system  in  the  Human  Machine 
Systems  Lab.  The  major  components  of  the  system  are: 

•  A  master-slave  teleoperation  system,  which  is  composed  of  a  pair  of  Phantom  haptic 
feedback  arms  and  has  stable  force  reflecting  control  (3  d.o.f.).  The  Phantom  arms  that  we 
use  are  identical  in  kinematics  and  are  mounted  (see  Figure  6-2)  such  that  an  operator 
standing  with  his  or  her  shoulder  under  the  robot  base  would  have  full  access  to  the  regular 
range  of  human  arm  motion.  The  phantom  arms  have  a  very  low  inertia  structure  which 
implies  ignorable  uncertainty  of  robot  dynamic  parameters  and  gives  us  an  advantage  in 
dynamic  control. 

•  Two  control  computers  which  are  used  to  control  the  master  robot  and  the  remote  slave 
robot  respectively.  The  corresponding  data  acquisition  cards  for  robot  control  are  installed 
in  each  machine. 

•  A  telecommunication  subsystem.  A  teleconference  equipment  set  is  used  in  the 
telesurgery  system  to  support  video  (image),  audio  transmission  and  control  data 
transmission  as  well,  through  three  ISDN  lines.  For  short  distance  communication,  the 
minimum  time  delay  due  to  image  compression  and  decompression  is  about  0.6  seconds 
(round  trip)  [Zy dacron,  1996].  There  are  two  computers  which  function  as  two 
telecommunication  terminals,  one  near  the  operator,  the  other  close  to  the  remote  patient. 

•  Two  data  transmission  links  between  control  computer  and  telecommunication 
terminal.  We  have  tested  two  different  types  of  data  links  between  two  PCs.  Ethernet  was 
one  option,  but  it  was  only  marginally  acceptable  for  force  reflection.  A  bidirectional 
parallel  dynamic  data  transmission  link  developed  in  our  lab  is  the  cheapest  and  fastest  way 
to  do  this  among  serial  port  data  transmission,  ethemet  transmission  and  other  options. 

•  Laparoscopic  surgery  simulator  (physical  task  environment)  with  surgical  tools 
[Ottensmeyer,  1996]. 

•  Mouse  pointer  overlay  device,  which  is  designed  especially  to  offer  a  human  machine 
interface  for  the  surgeon.  Basically,  the  mouse  icon  is  superposed  on  the  image  from  the 
patient,  and  thus  the  surgeon  can  send  instructions  visually  to  the  remote  assistant  by 


moving  the  mouse  icon  as  a  remote  pointer.  A  SGI  computer  with  video  overlay  hardware 
is  applied  for  this  purpose. 

•  Safety  Devices.  There  are  two  switches,  a  handle  switch  and  a  foot  switch  to  supply 
the  surgeon  and  assistant  with  appropriate  intervention,  respectively.  An  extra  electronic 
protection  card  is  installed  in  the  slave  robot’s  amplifier  box. 

Fig.  6-2  shows  a  picture  of  the  telesurgery  system. 

6.1.2  System  for  the  force  reflecting  control  experiment 

To  test  and  evaluate  the  designed  controller,  a  particular  system  configuration  was  designed 
(Figure  6-2)  with  a  beam  model. 

The  major  parts  are  the  same  as  Figure  6-1,  but  the  task  environment  has  been  changed 
to  a  physical  nonlinear  beam  model  which  is  supposed  to  simulate  the  patient’s  abdomen. 
In  addition,  a  virtual  delay  box  (computer  delay  buffer)  was  added  to  the  experimental 
system  for  the  purpose  of  adjusting  time  delays. 


6.2  Hardware  design 

(1)  Physical  beam  model  and  the  force  feedback  sensing  instrumentation  (see  Figure  6-3). 
This  subsystem  is  used  to  give  the  subject  visual  force  feedback  such  that  the  force  at  the 
end  effector  can  be  kept  constant  during  experiments.  It  can  also  provide  direct  force 
information  for  the  evaluation  of  controllers  in  test.  There  are  eight  strain  gages  in  the  force 
detection  circuit.  Four  gages  are  used  to  obtain  the  force  information  at  two  symmetric 
locations  on  the  beam,  and  four  other  gages  are  simply  applied  to  compensate  for  the 
temperature  effects.  The  additional  filters  and  amplifiers  can  supply  the  data  acquisition 
board  (A/D)  with  precise  signals  (Figure  6-4).  One  PC  (486)  is  used  to  process  the 
information  from  the  force  sensor  (strain  gages)  and  displays  the  force  response 
dynamically. 

(2)  Slave  electronic  protection  circuit.  This  circuit  provides  us  quick  torque  limit  protection 
for  the  three  joints  of  the  slave  robot. 

(3)  Bidirectional  parallel  port  dynamic  data  transmission  link.  The  newly  developed 
software  can  offer  the  user  a  cheap  way  to  do  fast  dynamic  data  transmission  between  two 
PCs  through  a  regular  parallel  port.  This  software  includes  data  security  checking  and 
control  signal  handshaking  protocol.  There  is  a  particular  connector  to  link  the  two  parallel 
ports  appropriately. 


6.3  Software  design 


All  the  software  has  been  developed  in  Borland  C++  (in  the  DOS  environment).  The  main 
program  can  supply  a  window  interface  for  the  user.  The  operator  can  select  the  particular 
control  subroutine  by  mouse  and  the  arrow  keys  on  the  keyboard,  and  the  time  delay  too. 
Currently  the  software  developed  for  the  project  has  the  following  functions: 

•  Interface  with  the  robot  control  ports,  encoder  ports. 

•  Time  delay  buffer  module. 

•  Three  different  versions  of  controller  subroutines  for  three  experiment  systems  (e.g. 
actual  master- virtual  slave  teleoperation  system,  master-slave  teleoperation  system  with 
beam  model,  and  master-slave  telerobotic  surgery  system). 

•  Data  transmission  software:  Bidirectional  ethemet  data  transmission  software  and 
bidirectional  parallel  port  data  transmission  software. 

•  System  pretest  software,  the  user  can  check  the  encoder  port  and  reset  the  control  port 
of  the  robots  and  the  encoder  neutral  position. 

•  Safety  on-line  protection  software:  position  protection  is  to  check  the  position  error 
between  the  master  and  the  slave,  and  velocity  protection  sets  a  velocity  limit  for  the 
robots. 

Listed  bellow  are  the  controller  subroutines  available  for  three  different  experiment  systems 
developed  for  the  project: 

(1)  master- virtual  slave  Teleoperation  system  control  software 
•FSC 

•Linear  PD  control 
•Supervisory  control 
•Passive  control 
•Predictive  display 

(2)  Master-slave  Teleoperation  system  control  software  with  beam  model 
•FSC 

•PD  control 
•Passive  control 

(3)  Master-slave  telesurgery  system  control  software 
•FSC 

•PD  Control 
•Passive  Control 


Slave 
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Figure  6-2:  Telerobotic  surgery  unit.  A  dummy  is  used  for  task  simulations. 


67 


Figure  6-3:  Force  feedback  sensing  instrument  of  the  beam 


Figure  6-4:  Electronic  circuits  for  force  signal  processing 


Chapter  7 

Experiment  Results  and  Analysis 

7.1  Dynamic  Responses. 

Typical  dynamic  responses  are  shown  in  this  section.  In  the  experiments,  a  physical  beam 
model  was  used  as  the  task  environment  (see  Figure  6-3).  There  are  three  different  human 
subject  experiments  conducted: 

1)  Position  drift  test  of  passive  control.  During  the  experiments,  the  human  subject  was 
asked  to  control  the  master  so  as  to  make  the  slave  contact  the  beam  with  mild  force,  then 
break  contact,  then  make  contact  again,  and  so  on.  Figure  7-1  shows  the  response  of 
passive  control  without  drift  compensation. 

2)  Passive  control  with  drift  compensation.  Both  the  stability  of  force  reflection  and  the 
position  tracking  accuracy  were  evaluated.  Figure  7-2  shows  the  responses  of  passive 
control  with  drift  compensation  ( 1  second  time  delay).  The  upper  figure  ( force  response ) 
was  recorded  when  the  human  subject  controlled  the  master  and  made  the  slave  contact  the 
beam  and  then  slid  on  the  beam  with  certain  force  from  a  start  point  to  the  other  end  of  the 
beam.  The  lower  figure  (position  response)  was  obtained  in  the  same  way  as  described  in 
1). 

3)  FSC  of  teleoperation.  The  subject  experiments  of  teleoperation  with  0.5  seconds  time 
delay  and  1  second  time  delay  were  conducted.  Figure  7-3a  and  Figure  7-3b  show  the 
responses  of  FSC.  The  experiments  were  carried  out  in  the  same  manner  as  in  2). 


7.2  Comparison  between  passive  control  and  FSC 

1)  Position  drifting  and  comparisons  between  Passive  Controllers  with /without  drift 
compensation. 

Figure  7-1  shows  the  position  responses  of  a  telerobotic  system  with  passive  control  but 
without  position  drift  compensation.  The  solid  line  is  the  response  of  master  robot,  and  the 
dash  line  is  the  response  of  slave  robot.  It  is  shown  clearly  that  the  slave  position  has  a 
drift,  which  increases  with  time.  Figure  7-1  shows  the  results  with  no  time  delay.  As 
explained  in  chapter  4,  the  force  term  in  the  slave  velocity  command  contributes  to  the  drift 
in  integration. 


The  responses  of  passive  control  with  drift  compensation  are  shown  in  Figure  7-2. 
From  the  responses  (with  1  second  time  delay),  the  teleoperation  system  with  passive 
control  is  stabilized,  and  with  drift  compensation  the  position  responses  have  good  tracking 
accuracy.  The  force  signal  is  also  smooth. 

2)  Comparison  between  FSC  and  Passive  Control.  The  responses  of  the  telerobotic 
system  with  FSC  are  shown  in  Figure  7-3.  The  system  with  FSC  is  stable  and  it  has  good 
position  tracking  accuracy.  Comparing  with  the  passive  control  approach,  The  telerobotic 
system  with  FSC  has  less  damping  factor,  and  hence  dissipates  less  operator’s  energy.  For 
the  passive  control  approach  with  drift  compensation  needs  a  position  scaling  factor  which 
is  equal  to  2. 
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Figure  7-1:  Responses  of  passive  control 1 
master  position  response,  and  the  dashed  line 
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Figure  7-2:  Responses  of  passive  control  with  drift  compensation  (1  second  time  delay ). 

Upper  figure  is  force  feedback  from  beam  model,  lower  figure  is  master  and  slave  position.  4 

Solid  line  is  master,  and  dashed  line  is  slave. 
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Figure  7-3  a:  Responses  of  FSC  with  0.5  seconds  time  delay.  Upper  figure  is  force  feedback 
from  beam  model,  lower  figure  is  master  and  slave  position.  Solid  line  is  master,  and  dashed 
line  is  slave. 
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Figure  7-3b:  Resp< 
from  beam  model, 
line  is  slave. 
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Chapter  8 

Conclusions  and  Future  Research 


8.1  Contributions 

In  this  thesis,  the  contributions  can  be  summarized  as  follows: 

1)  A  Novel  stable  fuzzy  control  approach,  FSC,  is  proposed  based  on  the  extension  of 
sliding  control.  This  design  provides  direct  stable  control  rule  tuning  in  the  phase  plane. 

2)  FSC  has  been  applied  in  stable  force  reflection  control  of  teleoperation  for  the  first 
time. 

3)  In  passive  control,  a  modified  approach  is  presented  to  solve  the  position  drift 
problem. 

4)  Experimental  hardware  and  software  design  and  implementations  have  resulted 

5)  FSC  Approach  is  better  than  the  conservative  passive  control  approach  in  that  it  saves 
human  operator’s  energy  because  passive  control  has  big  damping  components  and  make 
the  operator  fatigued. 


8.2  Future  research 

Telerobotics  is  a  very  broad  area  which  covers  artificial  intelligence,  control,  robot  design, 
communication,  human-machine  interaction,  etc..  There  are  many  practical  and  theoretical 
topics  to  be  investigated.  This  thesis  emphasizes  the  theoretical  and  experimental  study  of 
designing  stable  force  reflecting  controllers.  Related  to  this,  the  following  works  are 
suggested  to  continue  in  the  near  future: 

1)  Since  the  passive  control  approaches  are  based  on  linear  network  theory,  and 
nonlinearity  is  not  taken  into  account  during  the  controller  design.  It  is  worthwhile  to 
evaluate  the  control  performance  with  respect  to  different  nonlinear  tasks.  FSC  is  expected 
to  have  better  performance  in  dealing  with  a  nonlinearity.  It  is  interesting  to  compare  both 
with  respect  to  different  tasks. 

2)  In  this  thesis,  the  typical  dynamic  responses  give  us  an  objective  evaluation  of 
controllers,  but  for  overall  performance  evaluation  one  may  need  to  conduct  more 
subjective  evaluation  to  compare  the  energy  consumption,  the  fatiguing  factor  and  so  on. 


3)  There  is  continuing  need  to  develop  force  reflecting  surgical  tools.  It  is  believed  that 
force  feedback  in  surgical  tool  control  is  important  [Revetta,1996].  In  our  research,  the 
surgical  tools  are  only  position  drivable. 

4)  Adding  supervisory  control  mode  would  allow  the  user  to  switch  to  a  different  set  of 
control  parameters  for  different  complex  tasks. 

5)  Telepalpation  should  be  expected  in  this  context. 

6)  It  is  necessary  to  investigate  how  human  arm  locomotion  affects  the  performance  of 
teleoperation. 


Appendix  A 

The  Small  Gain  Theorem 


Generally,  the  size  of  a  linear  map  is  called  the  operator  norm  induced  by  a  vector  norm. 
Namely  the  magnification  power  of  a  system  H  on  a  set  of  input  vectors  u  is  the  norm  of 
the  output  vectors  H  ■  u  devided  by  the  norm  of  the  input  vector.  Then  the  gain  (operator 
norm )  of  the  system  H  induced  by  the  input  u  is  the  maximum  magnification  power  that 
the  system  H  can  exert  on  the  set  of  input  vector  u: 


|| //fl  =  sup  sup 
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(A.1) 
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For  a  SISO  system,  the  operator  norm  induced  by  the  set  of  energy-bounded  input  is 
simplified  into 


|H|  =  sup|ffO'm)|  (A.2) 

l o 

Thus,  the  operator  norm  of  a  LTI,  SISO  system  is  equivalent  to  the  maximum  amplitude  of 
the  transfer  function  of  the  system. 


Figure  A- 1 :  The  block  diagram  of  a  feedback  system 


In  Figure  A-l,  Hi  and  H1  can  be  any  kind  of  transfer  functions  (e.g.  scattering  matrices, 

impedance  matrices,  etc.)  of  a  feedback  system.  The  system  stability  is  ensured  if 
(/  +  #j#2)  remains  non-singular  at  all  frequencies,  where  H2  is  either  a  feedback 

controller  or  a  system  uncertainty. 

The  Small  Gain  Theorem: 

If  the  gains  (induced  norms),  which  are  chosen  in  accordance  with  the  input  vectors,  of 
Hl  and  H2  satisfies 

|H,M«2I<1  (A-3) 

then  the  transfer  function  (/  +  H{H2 )  will  remain  non-singular  and  the  system 
(/  +  HxH2  will  be  BIBO  stable. 

Corollary: 

Given  that  H  is  small-gain  but  otherwise  unknown,  the  system  (/  +  H]H2 is  stable 
iff  Hx  is  strictly  small-gain. 


Appendix  B 

Scattering  Theory  and  Passive  Control 

For  a  Two  port  circuit,  the  scattering  operator  S  is  defined  by, 

/  -  v  =  S(f  +  v)  (B.l) 


where, 
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Figure  B-l:  A  two  port  circuit 

In  the  frequency  domain,  the  scattering  operator  S  can  be  expressed  as  a  scattering 
matrix  S(s)  ( assuming  LTI  system),  where 


f(s)-v(s)  =  S(s)[f(s)  +  v(s)] 


(B.2) 


Define  the  hybrid  matrix  for  the  above  2-port  circuit. 
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Then,  one  can  derive  the  scattering  operator. 


S(s)  = 


T 
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[H(s)  -  /][tf(5)  +  7]-1 


(B.4) 


The  scattering  theorem: 

A  system  is  strictly  passive  iff  ||S|[  <  1 


where  ||5||  =  sup  X^^s(jd))*  s(j(0)^ 
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For  a  communication  link  with  a  transmission  delay  T  (second),  we  have 
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||S||  =  sup  X^s(jco)*  S(jCD)^ 

<0 

(B.10) 

=  sup(|tan(6)r|  +  |sec(u)7|)  =  00 

Therefore  the  communication  link  is  not  passive. 

0  e~sT1 

By  choosing  S  —  ,  and  considering  the  scaling  factor  B,  the  passive 
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control  can  be  derived, 
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Appendix  c 

A  Physical  Beam  Model 


Figure  C-l  A  physical  beam  model 

From  mechanics  of  material,  we  can  obtain  the  equation: 

s_Px2(L-x)2 
3  LEI 


(C.l) 


where  E:  Young’s  module 

I:  moment  of  inertia  of  a  plane  area 
L:  length  of  the  beam 
S  :  deflection 

P:  force  exerted  on  the  beam 

Therefore,  P  is  a  nonlinear  function  of  the  tangential  position  and  is  linear  to  the  normal 
displacement. 
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Chapter  1 


Introduction 

During  the  last  century  we  have  seen  an  extraordinary  change  in  the  way  we 
practice  medicine,  for  the  most  part  fueled  by  a  rapid  change  in  technology.  In  many 
cases,  technology  has  been  developed  for  an  application  outside  medicine,  but  has  then 
been  successfully  modified,  transferred  and  then  applied  to  the  solution  of  a  medical 
problem.  This  technology  transfer  is  now  being  studied  in  the  field  of  telerobotics  and  its 
application  to  inspection,  palpation,  manipulation  and  intervention  is  being  considered. 

Our  research  effort  focuses  on  two  interrelated  problems: 

1) .  Cooperative  manipulation  between  a  paramedic  local  to  the  patient  and 

a  surgeon  operating  through  a  telerobot. 

2) .  Coping  with  time  delay  in  both  visual  and  force  feedback. 

1.1  Background  and  Significance 

There  are  many  situations  where  an  acutely  and/or  critically  ill  patient  needs  to  be 
stabilized  and  cared  for  during  the  triage  and  transport  to  a  medical  center.  In  the  optimal 
case,  a  physician  who  would  be  riding  in  the  ambulance  would  perform  the  initial  care  of 
the  patient.  Due  to  manpower  constraints  this  is  not  possible,  and  necessary  care  in 
remote  locations  might  have  to  be  provided  by  individuals  who  have  minimal  or  no 
patient  management  skills.  Because  of  this,  no  major  diagnostic  or  therapeutic 
interventions  are  currently  possible  during  the  initial  triage  at  the  accident  site,  or  during 
the  subsequent  transport  to  the  major  medical  center.  We  are  proposing  a  telemedical 
acute  care  consultant  system  that  addresses  this  problem  in  an  effort  to  improve  patient 
morbidity  and  mortality  in  these  clinically  suboptimal  situations. 

Though  the  use  of  communication  systems  in  medicine  is  not  new,  the  use  of  a 
high  bandwidth  audio  and  video  link  has  recently  been  applied  to  the  diagnosis  and 
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treatment  of  the  patient.  Still,  there  are  numerous  situations  where  the  bandwidth  is  not 
sufficient  for  the  simultaneous  transmission  of  teleoperator  control  signals,  real  time 
video  signals  and  audio  signals.  In  such  a  case,  it  is  common  to  employ  codec 
(compression  and  decompression)  boards  for  the  transmission  of  the  video  signals. 
Standard  videoconferencing  equipment  over  ISDN  and  fractional  T3  phone  lines 
currently  use  this  technology  for  the  video  conferencing  calls.  This  video  compression 
and  decompression  imposes  a  time  delay,  and  combined  with  the  time  delay  imposed  by 
the  communication  system  produces  a  total  time  delay  that  adversely  affects  the  task 
completion  time  of  currently  proposed  telesurgical  systems  (explained  in  more  detail  in 
our  previous  report,  Human  Factors  in  Tele-Inspection  and  Tele-Surgery:  Cooperative 
Manipulation  and  Time  Delay,  dated  9  October  1995). 

Studies  by  Ferrell  and  Sheridan  (1967)  and  Black  (1970)  showed  that 
teleoperators  start  to  use  a  “move  and  wait”  strategy  as  the  closed  loop  time  delay 
increases  above  0.1  seconds.  In  our  first  series  of  experiments  (report  cited  above),  we 
found  that  the  synchronous  transmission  of  video  and  controller  signals  (our  model  of 
600  ms  video  delay  plus  a  600  ms  communication  delay)  resulted  in  a  total  round  trip 
time  delay  that  seriously  affected  task  performance  time  and  resulted  in  a  “move  and 
wait”  strategy  when  the  surgeon  operated  the  laparoscopic  tools.  In  this  current  set  of 
experiments  we  explore  the  hypothesis  that  sending  the  signals  from  a  remote 
telesurgical  setup  asynchronously  (sending  the  controller  signals  ahead  of  the  video 
signals,  which  were  delayed  because  of  the  compression  /  decompression  it  required) 
would  improve  controller  stability  and  possibly  favorably  affect  task  performance  by  the 
surgical  team. 

1.2  Technical  Terminology 

1.2.1  Cooperative  Tele-manipulation 

Cooperative  tele-manipulation  is  defined  as  the  interaction  (physical  manipulation 
and  audio-visual  communication)  between  individuals  working  toward  a  common 
objective.  Three  factors  are  important  in  cooperative  manipulation:  objective, 
communication,  and  manipulation: 
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Objective 

The  objective  of  cooperative  tele-manipulation  in  this  case  is  initial  diagnosis  and 
surgical  treatment  of  trauma  patients  during  triage  and  transport  to  the  medical  center. 

In  proposed  telesurgical  systems  the  ambulance  /  trauma  pod  will  be  operated  by 
two  individuals.  One  individual  will  be  responsible  for  driving  the  ambulance,  while  the 
other  will  assist  the  surgeon  operating  through  a  telemanipulator  in  the  care  of  the 
patient.  There  are  five  major  elements  in  this  system:  the  diagnostic  /  therapeutic  task 
(including  the  patient),  the  remote  surgeon,  the  assistant,  the  telemanipulator  and  tools, 
and  the  communication  channel  (defined  by  the  time  delay). 

The  assistant  will  be  a  paramedic  who  will  have  some  experience  with  trauma 
patients,  but  will  not  have  the  diagnostic  ability,  knowledge  base,  surgical  skill  or 
medical  decision-making  ability  of  the  surgeon. 

To  our  knowledge,  no  one  has  studied  the  interaction  between  a  human  and 
telemanipulator  controlled  by  another  human  remote  to  the  site,  both  collaborating  on  the 
same  sensory  and  manipulation  task. 

Communication 

Because  the  interaction  of  interest  is  between  a  paramedic  local  to  the  patient  site 
and  a  surgeon  at  a  remote  medical  center,  communication  is  via  an  audio-video  link  that 
unavoidably  has  some  amount  of  time  delay.  In  the  experimental  setup  for  the  Phantom 
arms,  a  digital  buffer  will  achieve  the  time  delay,  while  a  hardware  device  performs  the 
audio  and  video  delay. 

Three  classes  of  signals  are  sent  over  the  communication  link: 

1.  Telemanipulator  control  data.  The  output  (control  signals)  from  one 
telemanipulator  is  sent  over  the  communications  link  to  the  second  matched 
telemanipulator. 

2.  Video  images.  The  video  signals  from  the  laparoscope  (CCD  camera) 
are  compressed,  sent  over  a  fractional  T3  telephone  link  (3  multiplexed  ISDN  lines),  and 
then  decompressed  for  viewing. 

3.  Audio  signals.  This  includes  the  audio  from  the  microphone  connected 
to  the  assistant. 
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Manipulation 

The  diagnostic  and  therapeutic  tools  used  are  the  stethoscope,  DPL  cannula, 
laparoscopic  surgiports,  laparoscope,  and  laparoscopic  tools  (forceps,  shears,  blunt  probe, 
electrocautery,  and  clips). 

1.2.2  Synchronous  and  Asynchronous  Transmission 

In  our  first  set  of  experiments  we  buffered  the  control  signals  while  the  video 
signal  was  being  processed,  then  sent  both  video  and  controller  signals  simultaneously 
(synchronous)  over  the  transmission  link.  In  our  second  set  of  experiments  we 
immediately  sent  the  control  signals  over  the  transmission  link  without  waiting  for  the 
video  signals  (which  needed  0.6  second  round  trip  for  the  video  compression- 
decompression)  which  resulted  in  asynchronous  transmission  of  video  and  teleoperator 
control  signals. 

1.3  Medical  Terminology 

1.3.1  Trauma 

An  important  aspect  of  trauma  management  is  early  stabilization,  diagnosis  and 
surgical  intervention.  We  foresee  telediagnostic/telesurgery  systems  being  used  in  the 
diagnosis  and  the  initial  therapeutic  care  of  the  patient  with  blunt  and  penetrating 
abdominal  wounds.  An  initial  goal  is  the  stabilization  of  the  patient  by  medical  and 
preliminary  surgical  control  of  bleeding.  Once  this  is  accomplished,  effort  should  then 
center  on  the  judicious  use  of  available  investigative  techniques  to  determine  the  need  for 
operative  intervention.  By  this  approach  early  intervention  can  improve  the  morbidity 
and  mortality  associated  in  trauma. 

1.3.2  Telesurgery 

A  telediagnostic  communications  link  consists  of  a  (ideally  two  way)  closed- 
circuit  video  system,  where  the  medical  care  consultant  can  pan/tilt/zoom  a  remote 
camera  to  observe  the  patient  on  a  video  monitor  (and  ideally  the  local  medical  care 
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provider  with  the  patient)  with  a  synchronized  audio  link.  An  augmented  system  allows 
the  physician  to  remotely  position  surgical  instruments,  receive  tactile  and  kinesthetic 
feedback,  and  with  the  manipulative  assistance  of  the  local  provider,  perform  diagnostic 
and  preliminary  surgical  intervention  on  the  patient. 

The  type  of  telesurgical  link  used  is  dependent  on  the  communication  links 
available  at  the  site.  Alternatives  include  T1  (1.544  Megabits/sec),  fractional  T1  (0.384 
Megabits/sec),  T3  (45  Megabits/sec),  satellite  transmission,  and  microwave  link. 

There  are  various  barriers  to  implementation  of  telemedical  systems.  Foremost 
are  the  issue  of  liability  for  the  surgeon,  the  paramedic,  and  the  manufacturer  of  the 
equipment.  There  are  also  the  high  communication  costs  associated  with  long  distance 
telecommunication  rates  or  satellite  time,  in  addition  to  the  initial  costs  of  improving 
outdated  equipment  and  phone  lines.  In  most  cases,  there  are  no  standards  of  care  or 
reimbursement  mechanisms  in  place  for  such  telesurgical  services. 

1.3.3  Minimally  Invasive  Diagnostic  Procedures 

Peritoneal  Lavage 

Diagnostic  peritoneal  lavage  (DPL)  continues  to  play  an  important  diagnostic  role 
in  the  evaluation  of  blunt  abdominal  trauma.  There  are  three  methods  of  performing  a 
DPL.  In  the  “open”  technique,  the  surgeon  first  makes  an  incision  through  the  skin, 
subcutaneous  tissue,  linea  alba  and  peritoneum  before  inserting  the  cannula  into  the 
peritoneal  cavity  under  direct  vision.  In  the  “semi-open”  technique,  an  incision  is  made 
through  the  skin,  subcutaneous  tissue,  and  linea  alba  before  the  cannula  is  inserted 
through  the  peritoneum.  In  the  “closed”  technique,  the  cannula  is  inserted  through  all  the 
layers  of  the  abdomen  via  a  blind  stab.  Once  the  cannula  is  inserted,  aspiration  for  free 
intraperitoneal  blood  is  generally  recommended  followed  by  a  lavage  of  1000  cc  of 
normal  saline.  Peritoneal  fluid  is  then  aspirated  and  analyzed. 

Laparoscopy 

Dr.  Isaac  Ott  (a  Russian  Gynecologist)  first  reported  laparoscopy  in  1910;  he  used 
a  speculum  to  view  the  abdominal  viscera  in  a  dog.  Kelling,  in  1923,  reported  on  the  use 
of  a  cystoscope  for  the  same  procedure.  Gazzaniga  (Gazzaniga,  1976)  first  proposed  the 
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use  of  laparoscopy  in  the  diagnosis  of  abdominal  trauma.  The  introduction  of  the 
laparoscopic  cholecystectomy  8  years  later  started  the  latest  surge  in  interest  in 
laparoscopic  procedures. 

There  is  also  renewed  interest  in  the  use  of  laparoscopy  in  the  evaluation  of 
trauma  patients.  Earlier  studies  had  been  performed  before  its  widespread  use  in  general, 
orthopedic,  and  gynecological  surgery,  which  has  resulted  in  great  improvement  in 
instrumentation  and  video  quality.  Recently  Ivatury  (Ivatury  et  al,  1993)  evaluated  the 
use  of  laparoscopy  in  penetrating  abdominal  trauma,  and  found  that  the  diagnostic 
accuracy  was  excellent  for  hemoperitoneum,  solid  organ  injuries,  diaphragmatic 
lacerations,  and  retroperitoneal  hematomas,  but  it  had  a  poor  sensitivity  to  hollow  viscus 
injuries.  While  other  studies  support  this  contention,  further  investigation  of  laparoscopy 
for  evaluation  of  trauma  is  needed  to  assess  its  accuracy. 

In  the  experiment  reported  here  we  used  the  following  laparoscopic  equipment: 
DPL  cannula,  laparoscopic  surgiports,  laparoscope,  and  laparoscopic  tools  (forceps, 
shears,  blunt  probe,  electrocautery,  and  clips). 

1.4  Summary  of  this  thesis 

1.4.1  Research  Objectives 

The  long-term  goal  of  our  project  and  many  others  is  the  provision  of  practical 
telesurgical  systems  that  will  assist  the  medical  care  provider  (who  might  have  limited 
medical  care  experience)  in  the  management  of  a  medical  emergency  (including  surgical 
intervention)  at  a  remote  location.  Such  an  event  might  occur  in  both  civilian  and  non¬ 
civilian  situations  in  locations  such  as  battlefields,  natural  disasters,  research  stations  in 
the  Antarctic,  and  NASA  space  missions,  among  others. 

Our  specific  research  project  is  concerned  with  evaluating  telesurgery  from  a 
human  factors  engineering  viewpoint.  In  particular  it  focuses  on  two  interrelated 
problems: 

1) .  Cooperative  manipulation  between  a  paramedic  local  to  the  patient  and 

a  surgeon  operating  through  a  telerobot. 

2) .  Coping  with  time  delay  in  both  visual  and  force  feedback. 
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In  our  first  series  of  experiments  (previously  cited  report),  using  synchronous 
video  and  control  signals,  we  found  that  the  time  delay  made  operation  of  the 
laparoscopic  tools  by  the  surgeon  extremely  difficult. 

In  the  experiments  reported  here  we  asked  the  question:  Does  sending  the  signals 
from  a  remote  telesurgical  setup  asynchronously  (sending  the  controller  signals  ahead  of 
the  video  signals,  which  were  delayed  because  of  the  compression  /  decompression  it 
required)  improve  controller  stability  and  favorably  affect  task  performance  by  the 
surgical  team. 
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Chapter  2 


Research  Methodology 

In  section  2.1  we  will  explain  the  medical  issues  important  in  the  remote 
management  of  the  trauma  patient.  We  discuss  our  experimental  apparatus  in  section  2.2. 
In  section  2.3  we  identify  the  important  laparoscopic  surgical  procedures  applicable  to 
our  system,  followed  by  a  discussion  of  the  experimental  tasks  in  section  2.4. 


2.1  The  Telesurgical  System  to  be  Modeled 


In  the  proposed  telesurgical  system  the  ambulance  /  trauma  pod  will  be  operated 
by  two  individuals.  One  individual  will  be  responsible  for  driving  the  transport,  while  the 
other  will  assist  the  telemanipulator  in  the  care  of  the  patient.  This  is  similar  in  structure 
to  a  standard  ambulance  team  setup.  The  five  major  factors  mentioned  earlier  (the 
diagnostic  /  therapeutic  task,  the  remote  surgeon,  the  assistant,  the  telemanipulator  and 


tools,  and  the  communication  channel)  interact  as  illustrated  in  Figure  2.1. 


COMMUNICATION 

DELAY 


VIDEO  VIDEO 

DECOMPRESSION  COMPRESSION 

DELAY  DELAY 


LAPAROSCOPE 


Figure  2.1  Telesurgical  System 
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2.1.1  Telesurgical  System:  Diagnostic  /  Therapeutic  tasks 

The  diagnostic  task  is  a  function  of  the  current  state  of  the  patient.  Use  of  the 
telemanipulator  must  be  appropriate  to  the  nature  of  the  trauma,  yet  pose  a  minimum  of 
complexity  and  potential  harm  to  the  patient.  Laparoscopy,  a  minimally  invasive 
procedure,  is  safer  and  more  easily  performed  over  a  communication  line  than  an  open 
surgical  procedure.  Therefore  the  authors  feel  that  a  realistic  telediagnostic  goal  would 
be  to  assist  in  location  of  the  source  of  hemorrhage  from  a  blunt  and/or  penetrating 
traumatic  injury.  This  type  of  diagnosis  is  extremely  important  in  the  trauma  setting,  as 
the  majority  of  patient  deaths  on  the  battlefield  result  from  bleeding  prior  to  arrival  at  a 
surgical  facility. 

The  damage  caused  by  blunt  and/or  penetrating  injury  can  be  subdivided  into  two 
major  types,  internal  and  external.  The  assistant  can  identify  external  hemorrhagic 
injuries  with  or  without  assistance  by  the  remote  surgeon,  who  would  as  necessary  assist 
through  a  simple  video  hookup.  Because  of  the  location  of  the  injury,  internal 
hemorrhagic  injuries  are  more  difficult  to  evaluate  and  would  require  an  invasive 
procedure  for  diagnosis.  A  telemanipulator  controlled  laparoscope  and  /  or  blunt  probe  is 
a  minimally  invasive  system  that  would  be  a  useful  diagnostic  tool  for  internal  abdominal 
injuries.  The  tools  that  are  required  for  such  a  system  would  be  a  telemanipulator- 
controlled  laparoscope  and  a  telemanipulator-controlled  blunt  probe. 

The  therapeutic  task  in  the  care  of  the  trauma  patient  would  be  to  stabilize  the 
patient  before  and  during  transport  to  a  medical  center  where  more  definitive  diagnostic 
and  therapeutic  procedures  can  be  initiated.  In  the  case  of  the  trauma  patient,  an 
important  and  potentially  life  saving  therapeutic  maneuver  would  be  to  stop  the  bleeding 
from  external  and  internal  sites.  Again  a  surgical  telemanipulator  system  would  be  useful 
for  internal  hemorrhage  and  would  require  a  telemanipulator-controlled  laparoscope, 
telemanipulator-controlled  hemostat,  telemanipulator  controlled  clip  applicator,  and  a 
telemanipulator-controlled  scissors.  A  laparoscope  would  allow  the  surgeon  to  look  for 
internal  hemorrhage.  The  hemostat  would  be  used  as  a  blunt  probe  to  move  internal 
organs  and  to  temporarily  occlude  any  bleeding  vessels.  A  clip  applicator  would  be  used 
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to  further  stabilize  a  bleeding  vessel  and  allow  the  hemostat  to  be  removed  from  the 
abdomen.  The  scissors  would  be  used  to  cut  adhesions  for  a  better  view  of  the  surgical 
field. 

2.1.2  Surgeon 

The  role  of  the  surgeon  in  this  case  is  to  make  an  initial  diagnosis  and  decision  on 
what  therapeutic  intervention  is  needed  to  stabilize  the  patient.  The  surgeon  will  be 
constrained  by  the  fact  that  the  patient  is  at  a  remote  site,  but  will  have  the  services  of  the 
telemanipulator  and  the  assistant  who  is  with  the  patient.  The  surgeon  would  control  both 
the  master  end  of  the  telemanipulator  and  a  mouse  or  other  device  for  indicating 
locations  in  the  video  screen  of  the  assistant. 

2.1.3  Assistant 

The  assistant  would  most  likely  be  a  paramedic  who  will  have  some  experience 
with  trauma  patients,  but  would  not  have  the  diagnostic  ability,  knowledge  base,  surgical 
skill  or  medical  decision  making  ability  of  the  surgeon.  Depending  on  whether  the  setting 
is  civilian  or  military,  the  paramedic  for  each  case  is  usually  chosen  from  a  group  of 
similarly  qualified  individuals,  based  on  who  is  working  at  the  time  of  the  trauma,  and 
which  available  unit  is  physically  closest  to  the  patient.  Each  of  the  paramedics  will  have 
different  strengths  and  weaknesses,  based  on  innate  skills,  personality  and  training.  The 
assistant  controls  surgical  tools  not  under  control  by  the  surgeon,  as  indicated  in  Figure 
2.1. 

2.1.4  Telemanipulator  and  Tools 

The  telemanipulator  and  tools  are  important  in  enabling  the  surgeon  to  diagnose 
and  treat  the  trauma  patient  in  the  remote  environment.  As  mentioned  above,  a  minimally 
invasive  technique  using  a  laparoscope  and  laparoscopic  tools  is  probably  the  best 
method  of  achieving  our  diagnostic  and  therapeutic  goal  for  the  remote  trauma  patient. 
One  needs  to  be  able  to  control  a  laparoscope  for  viewing,  and  be  able  to  manipulate  a 
hemostat  (to  be  used  as  a  blunt  probe  and  to  grab  intra-abdominal  contents),  a  scissors 
and  a  gripper  /  clip  applicator  (for  clamping  the  bleeding  vessel). 
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2.1.5  Telesurgical  System:  Time  Delay 

In  a  telesurgical  system  one  is  faced  with  two  types  of  time  delays:  (1)  delay 
associated  with  the  transmission  of  the  signals  from  the  teleoperator  over  a 
communication  link,  and  (2)  the  delay  due  to  the  video  compression  and  decompression 
due  to  a  communication  bandwidth. 

The  transit  time  for  signals  sent  over  communication  channels  depends  on  the 
medium  (e.g.  ISDN,  satellite)  and  the  traffic  over  that  medium.  With  an  ISDN  setup,  the 
typical  round  trip  delays  are  in  the  order  of  600  ms  (Zydacron,  1996).  The  distance  can 
also  be  a  factor.  Researchers  reported  a  1.4  second  round  trip  delay  in  a  teleoperator 
connection  between  Tokyo  and  Washington,  D.C.  (Mitsuishi,  1995). 

There  is  also  a  delay  associated  with  the  video  compression  and  decompression 
needed  when  unlimited  bandwidth  is  not  available.  Current  video  conferencing 
equipment  that  uses  ISDN  lines  (fractional  T3,  which  is  three  multiplexed  ISDN  lines) 
requires  approximately  300  ms  for  the  video  compression  and  subsequent  decompression 
for  one  way  transmission.  The  components  of  these  delays  are  shown  in  Figure  2.1. 

2.2  Experimental  Apparatus 

Our  experimental  set-up  is  intended  to  model  the  telesurgical  system  described  in 
the  previous  section.  In  particular,  our  set-up  for  the  asynchronous  experiments  is  as 
shown  in  Figure  2-1.  The  components  were  designed  and  constructed  during  the  initial 
stages  of  our  project  and  explained  in  more  detail  in  our  previous  report,  Human  Factors 
in  Tele-Inspection  and  Tele-Surgery:  Cooperative  Manipulation  and  Time  Delay,  dated  9 
October  1995. 


2.2.1  Surgical  Tool  Handle 

In  order  to  make  the  working  environment  as  realistic  as  possible,  we  used 
modified  laparoscopic  tools  of  the  type  currently  being  used  in  the  operating  room.  The 
test  surgeon  operated  these  tool  handles  just  as  he  would  in  normal  laparoscopic  surgery. 
.  Our  modified  tool  handle  is  seen  in  Figure  2.2. 


15 


Figure  2.2:  Modified  tool  handle  for  master  telemanipulator 

2.2.2  Paired  Telemanipulators 

The  telemanipulators  used  were  paired  PHANToM  manipulators  that  were 
initially  developed  as  a  force-reflecting  interface.  One  of  the  phantoms,  operated  by  the 
test  surgeon,  was  used  as  a  master  on  which  the  tool  handle  was  fixed  (Figure  2.2).  The 
second  manipulator  (Figure  2.3)  operated  as  a  slave  (to  which  the  other  end  of  the 
laparoscopic  tool  was  attached).  The  control  software  used  was  developed  as  part  of  the 
project  (Hu  et  al,  1996)  and  is  an  attached  report. 
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Figure  2.3:  Modified  tool  tip  on  slave  telemanipulators 


2.2.3  Modified  Laparoscopic  Tools 

In  order  to  provide  as  much  realism  as  possible,  the  laparoscopic  tools  used  were 
actual  tools  used  in  the  operating  room  which  were  modified  for  our  use.  The  tool  handle 
was  detached  from  the  tip  and  attached  to  the  master  amnipulator.  Because  the 
PHANToM  telemanipulator  had  only  three  translational  degrees  of  freedom,  a  single 
rotational  degree  of  freedom  (around  the  tool  axis)  was  mechanized  (see  report  attached 
above)  and  is  shown  in  Figure  2.2.  For  the  slave  manipulator,  the  tool  tip  was  modified  to 
provide  a  roll  axis  and  is  attached  to  the  slave  phantom  using  the  gimble  wrist  end  of  the 
phantom.  This  attachment  to  the  slave  manipulator  is  shown  in  Figure  2.3  while  the 
various  modified  tools  and  their  linkages  are  shown  in  Figure  2.4. 
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Figure  2.4:  Modified  laparoscopic  tool 

The  modified  laparoscope  and  the  various  tool  tips  used  in  our  experiments  are  shown  in 
Figure  2.5. 

2.2.4  Audio  /  Visual  System 

The  audio  /  visual  system  consisted  of  microphones  and  headsets,  cameras  and 
displays  and  a  two-way  communications  setup.  Thus  both  audio  and  video  were  available 
to  both  the  surgeon  and  assistant.  We  used  two  matched  headphone  /  boom  microphone 
systems  connected  through  a  pre-amp  /  amplifier  to  the  audio  delay  boards.  The  camera 
at  the  patient  end  was  integral  to  the  laparoscope  itself. 

An  audio  /  visual  delay  system  was  used  to  generate  the  audio  and  visual  delays 
to  model  the  transmission  line  and  video  compression  /  decompression  delays.  We  used  a 
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Prime  Image  A/V  mainframe  with  two  video  delay  boards  and  a  2  channel  audio  board 
that  was  connected  to  our  camera,  monitors  and  audio  communication  equipment. 

The  video  images  were  recorded  with  a  Video  Hi-8  recorder  and  routed  to  the  two 
video  monitors  and  VHS  recorder  through  a  Silicon  Graphics  Galileo  video  board. 


Figure  2.5:  Modified  Laparoscope  and  tool  tips 


2.2.5  Laparoscopic  Test  Bed 

We  used  the  same  surgical  laparoscopic  simulators  designed  and  constructed  for 
our  first  series  of  experiments.  The  simulator  used  for  our  experiments  was  modeled  after 
laparoscopic  training  simulators  (Bailey  et  al,  1991  and  Muntzer,  1992)  which  has  four 
surgical  incision  sites  for  trocar  insertion  and  has  an  opaque  side  wall  for  recording  the 
experiments  with  a  camcorder.  This  is  shown  in  Figures  2.6. 
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Figure  2.6:  Exterior  view  of  simulator 

We  also  modified  a  resuscitation  mannequin  for  a  more  visually  realistic  laparoscopic 
model  for  demonstration  purposes  (Figure  2.7). 

2.2.6  Video  Overlay  System 

In  our  experimental  setup,  the  surgeon  operated  a  mouse  pointer  that  was 
superimposed  over  the  image  from  the  surgical  site  and  was  used  by  the  surgeon  as  a 
visual  tool  to  direct  the  actions  of  the  assistant.  This  video  overlay  was  generated  by  a 
Silicon  Graphics  Indigo2  Extreme  workstation  using  a  Galileo  video  card. 
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Figure  2.7:  Demonstration  simulator 


2.3  Normal  Laparoscopic  Surgical  Procedure 

2.3.1  Identification  of  “Abdominal  Contents” 

The  laparoscope  is  manipulated  inside  the  peritoneal  cavity  where  the  contents 
are  visually  inspected  for  signs  of  damage.  If  a  surgically  correctable  problem  is 
identified  then  the  procedure  continues  with  insertion  of  additional  surgiports  for  the 
laparoscopic  tools.  The  hemostat  is  used  to  move  and  manipulate  the  abdominal  organs 
in  order  to  give  the  physician  a  better  view  of  the  abdominal  contents.  The  ability  to 
move  and  view  contents  of  the  simulator  was  studied  and  presented  in  the  attached 
report. 
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2.3.2  Insertion  of  Surgiports  for  the  Laparoscopic  Tools 

Initially  two  additional  surgiports  are  inserted  with  visual  inspection  of  the 
insertion  sites  from  within  the  peritoneal  cavity  using  the  laparoscope.  This  is  to  prevent 
trauma  to  the  peritoneal  contents  from  the  surgiport  trocars. 

2.3.3  Insertion  and  Removal  of  Laparoscopic  Tools 

The  necessary  laparoscopic  tools  are  inserted  into  the  surgiport  under  visual 
observation. 

2.3.4  Visual  Localization  of  “Bleeding  Site” 

Initially  three  hands  are  utilized  for  the  identification  of  the  bleeding  site.  One 
hand  operates  the  laparoscope,  a  second  hand  operates  the  blunt  probe,  while  the  third 
hand  operates  the  forceps.  The  surgeon  has  the  option  of  quickly  switching  control  to  the 
tool  that  he  wants  to  manipulate  (foot  switch),  which  gives  him  better  control  over  the 
procedure. 

2.3.5  Blunt  Dissection  to  Isolate  “Bleeding  Site” 

Once  the  site  of  bleeding  is  localized,  then  surgical  manipulation  of  the  site  is 
needed  for  the  isolation  of  the  bleeding  vessel.  This  is  to  insure  that  important  structures 
(e.g.  nerves)  adjacent  to  the  bleeding  vessels  are  not  inadvertently  damaged  during 
permanent  control  of  the  bleeding  vessel.  Initial  control  of  bleeding  is  necessary  and  is 
temporarily  achieved  with  a  forceps  applied  to  the  general  area. 

2.3.6  Permanent  Control  of  Bleeding 

If  the  bleeding  vessel  is  small,  then  the  vessel  can  be  cauterized  electrically.  The 
electrocautery  is  applied  to  the  forceps  that  held  the  bleeding  vessel.  If  the  vessel  is  large, 
then  surgical  clips  are  applied  to  mechanically  occlude  the  bleeding  vessel.  This  is 
standard  surgical  practice  when  working  with  blood  vessels  that  are  no  longer  needed  or 
are  bleeding  and  permanent  control  of  the  bleeding  vessel  is  necessary.  If  the  vessel  is  an 
artery  that  is  important  for  blood  supply  to  an  organ  or  region,  then  the  vessel  would 
initially  be  occluded  for  control  and  later  repaired  in  a  more  definitive  procedure. 
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2.4  Experimental  Surgical  Tasks  Used 

2.4.1  Grasp  and  Transfer  Experiments 

This  experiment  evaluated  the  ability  of  the  surgical  team  to  control  the  hemostat 
and  to  work  together  to  perform  a  task.  In  this  task  the  assistant  picked  up  a  first  paper 
clip  with  the  hemostat,  passed  it  to  the  person  operating  the  second  hemostat,  who  then 
deposited  it  into  the  second  cup  (Figure  2.8).  This  proceeded  until  all  six  clips  were 
transferred  from  the  first  to  the  second  cup.  The  performance  time  was  defined  as  the 
time  it  takes  for  the  surgeon  -  assistant  team  to  transfer  6  clips  from  one  cup  to  the  other. 


Figure  2.8:  Grasp  and  Transfer  Experiment 
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2.4.2  Hemostasis 

In  this  experiment  we  evaluated  the  use  of  the  clip  applicator  /  hemostat  and  the 
scissors  by  the  surgical  team  (surgeon  and  assistant).  We  staredt  with  a  model  of  the 
neurovascular  bundle  (a  nerve,  artery  and  vein  usually  travel  together)  made  up  of  tan, 
blue  and  red  rubber  bands.  If  a  blood  vessel  is  damaged  and  is  bleeding,  the  goal  of  the 
surgical  team  is  to  isolate  the  bleeding  vessel  without  damaging  the  adjacent  nerve  and 
blood  vessel.  Once  the  bleeding  vessel  is  isolated  and  the  bleeding  controlled  it  is  clipped 
(stapled)  shut  and  the  redundant  vessel  is  cut.  The  task  completion  time  was  measured  as 
the  time  it  took  the  team  to  correctly  place  the  hemostat  /  clipper  completely  around  the 
related  vessel  (with  the  exclusion  of  adjacent  structures),  clip  off  the  vessel,  and  then  to 
cut  the  redundant  vessel  with  the  scissors.  This  experiment  is  shown  in  Figure  2.9. 
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2.5  Experimental  Subjects 

The  combination  of  two  major  experimental  setups,  four  surgical  tools  and  six 
combinations  of  time  delays  resulted  in  thirty  experiments  that  were  recorded  for  each  of 
the  six  subjects.  The  test  surgeon  throughout  the  experiments  was  an  experienced 
emergency  room  physician  and  anesthesiologist.  The  assistants  were  engineering 
graduate  students.  These  were  the  same  individuals  that  were  tested  in  the  first  set  of 
experiments  run  the  previous  year. 

2.6  Data  Recording 

The  experiments  were  recorded  to  a  VCR  deck  using  a  high  fidelity  Sony 
Camcorder.  The  author  later  reviewed  these  videotapes  and  the  task  completion  times 
were  recorded  for  each  of  the  individual  experiments. 
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Chapter  3 


Results  and  Discussion 

The  results  for  the  individual  tool  combinations  for  each  experiment  are  presented 
and  discussed  in  the  following  sections  of  this  chapter,  Figures  3.1  -3.4. 

In  general,  we  found  that  when  the  surgeon  operated  the  laparoscope  there  was  a 
slight  improvement  in  task  completion  time  when  the  task  was  performed  with 
asynchronous  (as  compared  to  synchronous)  video  and  control  feedback  signals.  When 
the  surgeon  used  any  of  the  other  tools,  we  found  a  dramatic  improvement  in  task 
completion  time  when  the  signals  were  sent  asynchronously,  for  both  the  0.6  second  and 
1.2  second  delays.  A  summary  of  the  average  task  completion  time  for  each  of  the 
experiments  and  tool  combination  is  presented  in  the  Appendix. 

3.1  Grasp  and  Transfer  Experiments 

There  were  two  physician  /  assistant  tool  combinations  for  this  experiment.  In  the 
first  scenario,  the  physician  operated  the  scope  through  the  teleoperator  while  the 
assistant  operated  the  two  hemostats.  In  the  second  scenario,  the  surgeon  operated  one  of 
the  hemostats  while  the  assistant  operated  the  laparoscope  and  the  other  hemostat.  As  in 
our  first  series  of  experiments,  we  found  that  task  completion  time  is  lowest  (better 
performance)  when  the  surgeon  operated  the  laparoscope  rather  than  operating  one  of  the 
hemostats.  We  found  this  to  be  the  case  in  all  six  experiments  in  which  we  varied  the 
time  delays  for  both  the  video  signals  and  the  controller  signals. 

In  our  second  set  of  experiments  we  studied  the  effect  of  asynchronous  video  and 
control  signal  transmission  on  task  completion  time  with  various  time  delays.  Our  results, 
which  are  presented  below  for  each  time  delay,  surprisingly  showed  a  significant 
improvement  in  task  completion  time  with  asynchronous  (as  compared  to  synchronous) 
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signal  transmission.  Apparently  it  was  better  to  have  force  feedback  sooner  even  if  it 
meant  it  was  not  synchronized  with  the  video.  A  detailed  explanation  follows: 

3.1.1  Experiments  with  no  transmission  time  delay  in  telemanipulation 

In  the  experiments  with  no  transmission  time  delay  the  controller  signals  were 
transmitted  between  teleoperators  without  any  delay  while  the  video  signals  were 
transmitted  with  a  0.6  sec  round  trip  delay  (inherent  in  the  video  compression  and 
decompression  algorithms).  In  our  first  set  of  experiments  we  held  the  control  signals 
while  the  video  signal  was  processed  and  then  sent  both  video  and  controller  signals 
simultaneously  (synchronous)  over  the  transmission  link.  In  our  second  set  of 
experiments  we  immediately  sent  the  control  signals  over  the  transmission  link  without 
waiting  for  the  video  signals  (which  needed  0.6  sec  round  trip  for  the  video  compression- 
decompression)  which  resulted  in  the  asynchronous  transmission  of  video  and 
teleoperator  control  signals. 

Physician  operating  the  laparoscope 

In  this  experiment  the  physician  operated  the  laparoscope  using  the  teleoperator 
while  the  assistant  local  to  the  patient  operated  the  two  hemostats.  In  the  synchronous 
experiments  (both  video  and  controller  were  transmitted  together  with  a  0.6  sec  round 
trip  delay)  the  average  completion  time  was  16.0  seconds.  In  the  asynchronous 
experiments  (the  controller  signals  were  transmitted  with  no  time  delay  while  the  video 
signal  was  sent  with  a  0.6  sec  round  trip  delay)  we  had  an  average  completion  time  of 
13.2  seconds.  These  experimental  results  show  a  17%  improvement  in  task  completion 
time  with  the  asynchronous  transmission  and  are  shown  graphically  in  Figure  3.1. 

Physician  operating  the  hemostat 

We  then  repeated  the  experiments  with  the  physician  operating  one  of  the 
hemostats  using  the  teleoperator  while  the  assistant  local  to  the  patient  operated  the 
second  hemostat  and  the  laparoscope.  In  the  synchronous  experiments  (both  video  and 
controller  were  transmitted  together  with  a  0.6  sec  round  trip  delay)  the  average 
completion  time  was  33.2  seconds.  In  the  asynchronous  experiments  (the  controller 
signals  were  transmitted  with  no  time  delay  while  the  video  signal  was  sent  with  a  0.6  sec 
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round  trip  delay)  we  had  an  average  completion  time  of  20.5  seconds.  These 
experimental  results  show  a  38%  improvement  in  task  completion  time  with  the 
asynchronous  transmission  and  are  also  shown  in  Figure  3.1 


GRASP  &  TRANSFER  -  SCOPE  GRASP  &  TRANSFER  -  HEMOSTAT 

Experiment 


Figure  3.1.  Grasp  and  Transfer  Experiments:  Performance  times  for  asynchronous 
and  synchronous  signals  with  no  transmission  time  delay 

3.1.2  Experiments  with  a  0.6  second  transmission  time  delay  in  telemanipulation 

In  the  experiments  with  a  0.6  second  round  trip  transmission  time  delay  the 
controller  signals  were  transmitted  between  teleoperators  with  a  0.6  delay  while  the  video 
signals  were  transmitted  with  a  1.2  sec  round  trip  delay  (0.6  video  compression  and 
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decompression  time  delay  plus  the  0.6  second  transmission  time  delay).  In  our  first  set  of 
experiments  we  buffered  the  control  signals  while  the  video  signal  was  processed  and 
then  sent  both  video  and  controller  signals  simultaneously  (synchronous)  over  the 
transmission  link.  In  our  second  set  of  experiments  we  immediately  sent  the  control 
signals  over  the  transmission  link  without  waiting  for  the  video  signals  (which  needed 
0.6-sec  round  trip  for  the  video  compression-decompression)  which  resulted  in  the 
asynchronous  transmission  of  video  and  teleoperator  control  signals. 

Physician  operating  the  laparoscope 

In  this  experiment  the  physician  operated  the  laparoscope  using  the  teleoperator 
while  the  assistant  local  to  the  patient  operated  the  two  hemostats.  In  the  synchronous 
experiments  (both  video  and  controller  were  transmitted  together  with  a  1.2  sec  round 
trip  delay)  the  average  completion  time  was  15.1  seconds.  In  the  asynchronous 
experiments  (the  controller  signals  were  transmitted  with  a  0.6  second  time  delay  while 
the  video  signal  was  sent  with  a  1.2  second  round  trip  delay)  we  had  an  average 
completion  time  of  14.0  seconds.  These  experimental  results  show  a  6.4%  improvement 
in  task  completion  time  with  the  asynchronous  transmission  and  are  shown  graphically  in 
Figure  3.2. 

Physician  operating  the  hemostat 

We  then  repeated  the  experiments  with  the  physician  operating  one  of  the 
hemostats  using  the  teleoperator  while  the  assistant  local  to  the  patient  operated  the 
second  hemostat  and  the  laparoscope.  In  the  synchronous  experiments  (both  video  and 
controller  were  transmitted  together  with  a  1.2  sec  round  trip  delay)  the  average 
completion  time  was  43.5  seconds.  In  the  asynchronous  experiments  (the  controller 
signals  were  transmitted  with  a  0.6  second  time  delay  while  the  video  signal  was  sent 
with  a  1.2  second  round  trip  delay)  we  had  an  average  completion  time  of  30.1  seconds. 
These  experimental  results  show  a  31%  improvement  in  task  completion  time  with  the 
asynchronous  transmission  and  are  shown  graphically  in  Figure  3.2 
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Figure  3.2.  Grasp  and  Transfer  Experiments:  Performance  times  for  asynchronous 
and  synchronous  signals  with  a  0.6  second  transmission  time  delay 

3.2  Hemostasis  Experiments 

There  were  three  physician  /  assistant  tool  combinations  for  this  experiment,  a 
laparoscope,  clipper  /  hemostat,  and  a  scissors.  In  the  first  scenario,  the  physician 
operated  the  scope  through  the  teleoperator,  while  the  assistant  operated  the  clipper  and 
the  scissors.  In  the  second  scenario,  the  surgeon  operated  the  clipper  /  hemostat  while  the 
assistant  operated  the  laparoscope  and  the  scissors.  In  the  third  scenario  the  physician 
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operated  the  scissors  while  the  assistant  operated  the  laparoscope  and  the  clipper.  As  in 
our  first  series  of  experiments,  task  completion  time  is  lowest  (better  performance)  when 
the  surgeon  operated  the  laparoscope  rather  than  operating  one  of  the  other  tools.  We 
found  this  to  be  the  case  in  all  six  experiments  in  which  we  varied  the  time  delays  for 
both  the  video  signals  and  the  controller  signals. 

In  our  second  set  of  experiments  we  studied  the  effect  of  asynchronous  video  and 
control  signal  transmission  on  task  completion  time  with  various  time  delays.  Our  results, 
which  are  presented  below  for  each  time  delay,  also  showed  a  significant  improvement  in 
task  completion  time  with  asynchronous  (as  compared  to  synchronous)  signal 
transmission,  except  when  the  physician  operated  the  laparoscope. 

3.2.1  Experiments  with  no  transmission  time  delay  in  telemanipulation 

Physician  operating  the  laparoscope 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted 
together  with  a  0.6  sec  round  trip  delay)  the  average  completion  time  was  38.5  seconds. 
In  the  asynchronous  experiments  (the  controller  signals  were  transmitted  with  no  time 
delay  while  the  video  signal  was  sent  with  a  0.6  sec  round  trip  delay)  we  had  an  average 
completion  time  of  36.0  seconds.  These  experimental  results  show  a  negligible  difference 
in  task  completion  time  between  synchronous  and  asynchronous  transmission  and  are 
shown  graphically  in  Figure  3.3 

Physician  operating  the  hemostat  /  clipper 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted 
together  with  a  0.6  sec  round  trip  delay)  the  average  completion  time  was  103.1  seconds. 
In  the  asynchronous  experiments  (the  controller  signals  were  transmitted  with  no  time 
delay  while  the  video  signal  was  sent  with  a  0.6  sec  round  trip  delay)  we  had  an  average 
completion  time  of  42.2  seconds.  These  experimental  results  show  a  59%  improvement 
in  task  completion  time  with  the  asynchronous  transmission  and  are  shown  graphically  in 
Figure  3.3 


31 


GRASP  &  TRANSFER  -  SCOPE  GRASP  &  TRANSFER  -  HEMOSTAT 

Experiment 


Figure  3.3.  Hemostasis  Experiments:  Performance  times  for  asynchronous  and 
synchronous  signals  with  no  transmission  time  delay 

Physician  operating  the  scissors 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted 
together  with  a  0.6  sec  round  trip  delay)  the  average  completion  time  was  73.9  seconds. 
In  the  asynchronous  experiments  (the  controller  signals  were  transmitted  with  no  time 
delay  while  the  video  signal  was  sent  with  a  0.6  sec  round  trip  delay)  we  had  an  average 
completion  time  of  40.0  seconds.  These  experimental  results  show  a  46%  improvement 
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in  task  completion  time  with  the  asynchronous  transmission  and  are  shown  graphically  in 
Figure  3.3 

3.2.2  Experiments  with  a  0.6  second  transmission  time  delay  in  telemanipulation 

Physician  operating  the  laparoscope 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted 
together  with  a  1.2  sec  round  trip  delay)  the  average  completion  time  was  36.7  seconds. 
In  the  asynchronous  experiments  (the  controller  signals  were  transmitted  with  a  0.6 
second  time  delay  while  the  video  signal  was  sent  with  a  1.2  second  round  trip  delay)  we 
had  an  average  completion  time  of  35.9  seconds.  As  in  the  no  transmission  delay  case, 
these  experimental  results  show  a  negligible  difference  in  task  completion  time  between 
synchronous  and  asynchronous  transmission  and  are  shown  graphically  in  Figure  3.4 

Physician  operating  the  hemostat  /  clipper 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted  together  with 
a  1.2  sec  round  trip  delay)  the  average  completion  time  was  214.5  seconds.  In  the 
asynchronous  experiments  (the  controller  signals  were  transmitted  with  a  0.6  second  time 
delay  while  the  video  signal  was  sent  with  a  1 .2  second  round  trip  delay)  there  was  an 
average  completion  time  of  85.4  seconds.  These  experimental  results  show  a  60% 
improvement  in  task  completion  time  with  the  asynchronous  transmission  and  are  shown 
graphically  in  Figure  3.4 

Physician  operating  the  scissors 

In  the  synchronous  experiments  (both  video  and  controller  were  transmitted 
together  with  a  1.2  sec  round  trip  delay)  the  average  completion  time  was  113.4  seconds. 
In  the  asynchronous  experiments  (the  controller  signals  were  transmitted  with  a  0.6 
second  time  delay  while  the  video  signal  was  sent  with  a  1.2  second  round  trip  delay)  we 
had  an  average  completion  time  of  57.3  seconds.  These  experimental  results  show  a  49% 
improvement  in  task  completion  time  with  the  asynchronous  transmission  and  are  shown 
graphically  in  Figure  3.4 
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Figure  3.4.  Hemostasis  Experiments:  Performance  times  for  asynchronous  and 
synchronous  signals  with  a  0.6  second  transmission  time  delay 

3.3  Discussion 

In  this  second  series  of  experiments  in  our  telesurgery  project  we  attempted  to  test 
the  hypothesis  that  sending  the  signals  from  a  remote  telesurgical  setup  asynchronously 
(sending  the  controller  signals  ahead  of  the  video  signals,  which  were  delayed  because  of 


34 


the  compression  /  decompression  it  required)  would  improve  controller  stability  and 
favorably  affect  task  performance  by  the  surgical  team. 

We  had  very  interesting  results.  There  was  essentially  no  difference  in  task 
performance  between  the  synchronous  and  asynchronous  transmission  of  the  telesurgical 
signals  when  the  physician  operated  the  laparoscope.  But  we  found  a  significant 
improvement  (31%  to  60%)  in  task  completion  time  when  the  physician  operated  any  of 
the  laparotomy  tools  with  asynchronous  transmission  when  compared  to  synchronous 
transmission. 

The  result  when  the  physician  operated  the  laparoscope  can  be  easily  explained 
based  on  control  feedback.  Degradation  in  team  performance  resulting  from  the  effects  of 
time  delay  (controller  instability)  is  reduced  by  several  factors.  First,  because  the 
laparoscopic  output  is  an  entire  field,  the  person  controlling  the  laparoscope  needs  to  be 
less  sophisticated.  Secondly,  because  the  surgeon  knows  the  contents  of  the  surgical 
field,  and  what  task  sequence  needed  to  be  performed,  he/she  is  more  efficient  in 
minimizing  movements  (stabilizing  field  of  view)  of  the  laparoscope.  In  essence,  the 
control  of  the  laparoscope  was  fairly  passive  as  compared  to  the  operation  of  the 
laparoscopic  tools. 

The  significant  improvement  in  task  completion  time  (for  asynchronous  as 
contrasted  to  synchronous  transmission)  when  the  surgeon  operated  the  laparoscopic 
tools  was  unexpected.  Obviously,  the  advantages  of  asynchronous  transmission  more 
than  offset  the  disadvantage  of  asynchronous  transmission,  but  this  is  counter-intuitive. 
We  should  separately  evaluate  the  effects  of  asynchronous  transmission  and  then  put  the 
story  together  to  try  to  explain  our  results. 

The  main  advantage  with  asynchronous  transmission  probably  resulted  from  the 
increased  stability  of  the  telerobotic  manipulator  because  of  the  decreased  time  delay  that 
the  controller  saw.  In  the  case  of  synchronous  transmission  (Figure  2.1),  since  the  control 
signals  were  sent  with  the  video  signals,  the  controller  had  an  additional  delay  due  to  the 
time  it  took  to  compress  and  then  decompress  the  video  signals.  The  controller  delay  was 

Td  =  Dc  +  D+Dd 
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where:  Td  is  the  total  delay  seen  by  the  controller 
Dc  is  the  delay  due  to  video  compression 
D  is  the  delay  due  to  signal  transmission  (e.g.  phone  line) 

Dd  is  the  delay  due  to  video  decompression 

In  the  case  of  asynchronous  transmission  (Figure  2.1),  the  control  signal  was  sent  ahead 
of  the  video  signal  and  the  controller  delay  was  then: 

Td  =  D 

where:  Td  is  the  total  delay  seen  by  the  controller 

D  is  the  delay  due  to  signal  transmission  (e.g.  phone  line) 

So  we  see  that  the  controller  had  a  smaller  time  delay  in  the  asynchronous  mode,  which 
made  it  more  stable  and  easier  for  the  physician  to  operate  the  tools. 

The  main  disadvantage  with  asynchronous  transmission  was  that  the  operator  did 
not  receive  the  force  feedback  information  at  the  same  that  he  received  the  video  input. 
The  results  of  our  experiments  seem  to  suggest  that  the  improvement  in  performance 
because  of  the  more  stable  controller  more  than  offset  the  degradation  in  performance 
due  to  the  asynchronous  force  feedback  and  visual  image  seen  by  the  physician. 
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Chapter  4 


Conclusions  and  Future  Research 

4.1  Conclusions 

In  this  second  series  of  experiments  in  our  telesurgery  project  we  attempted  to  test 
the  hypothesis  that  sending  the  signals  from  a  remote  telesurgical  setup  asynchronously 
(sending  the  controller  signals  as  quickly  as  possible,  ahead  of  the  video  signals,  which 
are  necessarily  delayed  because  of  the  compression  /  decompression  required)  improves 
controller  stability  and  favorably  affects  task  performance. 

There  was  essentially  no  difference  in  task  performance  between  the  synchronous 
and  asynchronous  transmission  of  the  telesurgical  signals  when  the  physician  operated 
the  laparoscope  and  the  assistant  operated  the  laparotomy  tools.  But  with  asynchronous 
transmission  when  compared  to  synchronous  transmission  we  found  a  significant 
improvement  (31%  to  60%)  in  task  completion  time  when  the  physician  operated  any  of 
the  laparotomy  tools. 

This  is  an  important  finding  that  can  be  applied  to  any  situation  which  involves 
the  cooperative  actions  of  an  expert  operating  a  remote  telemanipulator  system  (that  is 
limited  by  bandwidth  considerations)  and  a  non-expert  person  operating  some  tools  at  the 
local  site  (where  the  telemanipulator  slave  is  working). 

If  the  expert  operating  the  telemanipulator  is  using  a  tool  that  requires  movement 
or  manipulation,  then  one  can  improve  task  completion  time  if  the  control  signal  and  the 
video  signals  are  sent  as  soon  as  possible  (asynchronous  to  one  another),  rather  than 
waiting  for  the  video  compression  /  decompression  and  then  sending  the  two  signals 
together. 
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4.2  Future  Research 


In  our  series  of  experiments  we  varied  the  time  delays,  surgical  tasks,  surgical 
tools  and  assistants  but  kept  the  surgical  expert  constant.  For  our  asynchronous 
experiments,  there  is  the  possibility  that  a  different  surgical  expert  might  have  a  problem 
(or  conversely,  have  an  easier  time)  with  the  asynchronous  force  feedback  and  visual 
inputs.  We  suggest  that  the  experiments  be  repeated  with  different  surgical  experts  to  test 
this  hypothesis. 
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SCOPE 

CLIPPER 

SCISSORS 

VIDEO  TIME  DELAY 

0.6 

1.2 

1.2 

1.2 

CONTROL  TIME  DELAY 

0.6 

1.2 

1.2 

1.2 

DATA 

113 

33 

584 

60 

SUBJECT  NICK 


EXPERIMENT  HEMOSTASIS  HEMOSTASIS  HEMOSTASIS 


CONDITION  TELEOPERATOR  TELEOPERATOR  TELEOPERATOR 


SURGEON  OPERATES  SCOPE  CLIPPER  SCISSORS 


VIDEO  TIME  DELAY  0.6  0.6  0.6 


CONTROL  TIME  DELAY  0  0  I  I  0 


MEAN 


VARIANCE 


27.40 


29.80 


47.67 


276.33 


32.67 


180.33 


SUBJECT 


EXPERIMENT 


CONDITION 
SURGEON  OPERATES 


VIDEO  TIME  DELAY 


CONTROL  TIME  DELAY 
DATA 


NICK  NICK  |  |  NICK  | 


HEMOSTASIS  HEMOSTASIS  HEMOSTASIS 


TELEOPERATOR  TELEOPERATOR  TELEOPERATOR 
SCOPE  ’  CLIPPER  ’  SCISSORS 


1.2  1.2  1.2 


0.6  0.6  0.6 


MEAN 

21.00 

68.00 

VARIANCE 

16.00 

1348.00 

SUBJECT 

JAY 

JAY 

JAY 

JAY 

EXPERIMENT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

CONDITION 

DIRECT 

DIRECT 

TELEOPERATOR 

TELEOPERATOR 

SURGEON  OPERATES 

SCOPE 

HEMOSTAT 

SCOPE 

HEMOSTAT 

VIDEO  TIME  DELAY 

NA 

NA 

0 

0 

CONTROL  TIME  DELAY 

NA 

NA 

0 

DATA 

HEHil 

24 

19 

25 

14 

20 

32 

13 

10 

36 

15 

10 

10 

10 

15 

20 

10 

31 

24 

7 

11 

14 

36 

13 

22 

17 

25 

10 

38 

9 

16 

11 

29 

13 

24 

14 

immsmmi 

12 

10 

13 

12 

8 

36 

ii 

MEAN 

23.83 

13.75 

19.25 

17.08 

VARIANCE 

65.37 

36.39 

106.93 

71.72 

HMMi 

IMiliilMiilM 

mm 

■■MBMMI 

■  II 

■1 

mmmamm 

■  II 

■1 

n  ■ 

timwmmswm 

SUBJECT 

JAY 

JAY 

JAY 

JAY 

EXPERIMENT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

CONDITION 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

SURGEON  OPERATES 

SCOPE 

HEMOSTAT 

SCOPE 

HEMOSTAT 

VIDEO  TIME  DELAY 

0.6 

0.6 

1.2 

1.2 

CONTROL  TIME  DELAY 

0.6 

0.6 

1.2 

1.2 

DATA 

19 

45 

10 

32 

12 

30 

26 

■EHI 

18 

27 

19 

■BHI 

15 

22 

14 

30 

19 

26 

48 

32 

12 

29 

16 

29 

21 

27 

10 

59 

26 

29 

30 

39 

14 

30 

11 

42 

8 

18 

36 

46 

17 

33 

10 

104 

28 

23 

12 

44 

MEAN 

17.42 

28.25 

20.17 

45.67 

VARIANCE 

33.54 

44.57 

152.15 

416.24 

61 


SUBJECT 

JAY 

JAY 

JAY 

JAY 

EXPERIMENT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

HEMOSTAT 

CONDITION 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

SURGEON  OPERATES 

SCOPE 

HEMOSTAT 

SCOPE 

HEMOSTAT 

VIDEO  TIME  DELAY 

0.6 

0.6 

1.2 

1.2 

CONTROL  TIME  DELAY 

0 

0 

0.6 

0.6 

DATA 

10 

20 

10 

20 

10 

15 

11 

25 

10 

12 

11 

19 

13 

14 

9 

18 

6 

14 

17 

25 

27 

19 

34 

55 

17 

12 

11 

41 

8 

46 

18 

41 

8 

20 

11 

18 

13 

33 

11 

27 

7 

48 

13 

43 

11 

36 

10 

65 

MEAN 

11.67 

24.08 

13.83 

33.08 

VARIANCE 

32.42 

173.72 

47.97 

246.81 

SUBJECT 

JAY 

JAY 

JAY 

JAY 

EXPERIMENT 

HEMOSTASIS 

HEMOSTASIS 

HEMOSTASIS 

HEMOSTASIS 

CONDITION 

DIRECT 

DIRECT 

DIRECT 

TELEOPERATOR 

SURGEON  OPERATES 

SCOPE 

CLIPPER 

SCISSORS 

SCOPE 

VIDEO  TIME  DELAY 

NA 

NA 

NA 

0 

CONTROL  TIME  DELAY 

NA 

NA 

NA 

0 

DATA 

32 

29 

37 

28 

28 

30 

38 

34 

32 

20 

62 

34 

36 

20 

33 

29 

30 

16 

28 

36 

MEAN 

31.60 

23.00 

42.50 

32.20 

VARIANCE 

8.80 

38.00 

173.67 

12.20 

62 


SUBJECT 

JAY 

JAY 

JAY 

JAY 

EXPERIMENT 


CONDITION 
SURGEON  OPERATES 


HEMOSTASIS 


HEMOSTASIS 


HEMOSTASIS 


HEMOSTASIS 


TELEOPERATOR  TELEOPERATOR  TELEOPERATOR  TELEOPERATOR 


CLIPPER 


SCISSORS 


SCOPE 


CLIPPER 


VIDEO  TIME  DELAY 

0 

0 

0.6 

CONTROL  TIME  DELAY 

0 

0 

0.6 

DATA 

20 

32 

46 

129 

21 

51 

29 

107 

18 

31 

29 

84 

27 

42 

35 

32 

29 

38 

MEAN 

23.60 

37.00 

35.40 

106.67 

VARIANCE 

33.30 

86.50 

50.30 

506.33 

SUBJECT 


EXPERIMENT 

HEMOSTASIS 

HEMOSTASIS 

HEMOSTASIS 

HEMOSTASIS 

CONDITION 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

SURGEON  OPERATES 

SCISSORS 

SCOPE 

CLIPPER 

SCISSORS 

VIDEO  TIME  DELAY 


CONTROL  TIME  DELAY 


DATA 


SUBJECT 


EXPERIMENT 

HEMOSTASIS 

HEMOSTASIS 

HEMOSTASIS 

CONDITION 

TELEOPERATOR 

TELEOPERATOR 

TELEOPERATOR 

SURGEON  OPERATES 

SCOPE 

CLIPPER 

SCISSORS 

VIDEO  TIME  DELAY 

0.6 

0.6 

0.6 

CONTROL  TIME  DELAY 

0 

0 

0 

DATA 

43 

55 

47 

25 

36 

34 

37 

44 

37 

54 

29 

MEAN 

37.60 

45.00 

39.33 

VARIANCE 


SUBJECT 


EXPERIMENT 


CONDITION 
SURGEON  OPERATES 


VIDEO  TIME  DELAY 


CONTROL  TIME  DELAY 

_____ 


HEMOSTASIS  HEMOSTASIS  HEMOSTASIS 


TELEOPERATOR  TELEOPERATOR  TELEOPERATOR 
SCOPE  CLIPPER  ’  SCISSORS 


!  1.2  1.2  1.2 


0.6  0.6  0.6 


MEAN 

34.67 

84.33 

68.33 

VARIANCE 

22.33 

581.33 

702.33 

JMT 

